* Increasing GNAT's heap @ 2013-11-12 11:09 Dmitry A. Kazakov 2013-11-12 13:26 ` Georg Bauhaus ` (3 more replies) 0 siblings, 4 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-12 11:09 UTC (permalink / raw) Does anybody know a way to increase the heap size available for GNAT? It crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. GNAT 4.6.4 under Debian, testing, 32-bit -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 11:09 Increasing GNAT's heap Dmitry A. Kazakov @ 2013-11-12 13:26 ` Georg Bauhaus 2013-11-12 14:00 ` Dmitry A. Kazakov 2013-11-12 22:03 ` Florian Weimer ` (2 subsequent siblings) 3 siblings, 1 reply; 29+ messages in thread From: Georg Bauhaus @ 2013-11-12 13:26 UTC (permalink / raw) On 12.11.13 12:09, Dmitry A. Kazakov wrote: > Does anybody know a way to increase the heap size available for GNAT? It > crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. This is not, I guess, about how in tasks, pragma Storage_Size (Dynamic_Value); affects storage available to objects in the task? The maximum possible value here seems bounded by System.Parameters.Size_Type. Do ulimit values have any effect? ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 13:26 ` Georg Bauhaus @ 2013-11-12 14:00 ` Dmitry A. Kazakov 2013-11-12 17:31 ` Paul Rubin 2013-11-12 19:30 ` Georg Bauhaus 0 siblings, 2 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-12 14:00 UTC (permalink / raw) On Tue, 12 Nov 2013 14:26:42 +0100, Georg Bauhaus wrote: > On 12.11.13 12:09, Dmitry A. Kazakov wrote: >> Does anybody know a way to increase the heap size available for GNAT? It >> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. > > Do ulimit values have any effect? $ ulimit unlimited -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 14:00 ` Dmitry A. Kazakov @ 2013-11-12 17:31 ` Paul Rubin 2013-11-13 8:00 ` Dmitry A. Kazakov 2013-11-12 19:30 ` Georg Bauhaus 1 sibling, 1 reply; 29+ messages in thread From: Paul Rubin @ 2013-11-12 17:31 UTC (permalink / raw) "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: > $ ulimit > unlimited Try "ulimit -a" and check the individual parameters. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 17:31 ` Paul Rubin @ 2013-11-13 8:00 ` Dmitry A. Kazakov 0 siblings, 0 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 8:00 UTC (permalink / raw) On Tue, 12 Nov 2013 09:31:20 -0800, Paul Rubin wrote: > "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >> $ ulimit >> unlimited > > Try "ulimit -a" and check the individual parameters. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 64846 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 64846 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 14:00 ` Dmitry A. Kazakov 2013-11-12 17:31 ` Paul Rubin @ 2013-11-12 19:30 ` Georg Bauhaus 1 sibling, 0 replies; 29+ messages in thread From: Georg Bauhaus @ 2013-11-12 19:30 UTC (permalink / raw) On 12.11.13 15:00, Dmitry A. Kazakov wrote: > On Tue, 12 Nov 2013 14:26:42 +0100, Georg Bauhaus wrote: > >> On 12.11.13 12:09, Dmitry A. Kazakov wrote: >>> Does anybody know a way to increase the heap size available for GNAT? It >>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >> >> Do ulimit values have any effect? > > $ ulimit > unlimited FWIW, I am finding it difficult to trigger something beyond "Storage_Error heap exhausted" (no bug, i.e.); SE seems kind of expected for some amounts of memory requested. The program below tries to make the memory manager commit, by forcing random inits of the memory at random positions. The program runs quite a while before either me interrupting it at 2**34 storage elements (Mac OS X 10.7.5), or some Linux VMs duly reporting STORAGE_ERROR while processing the memory for 2**32. Both systems are 64 bit, though, so I can't tell what 32 bit gives as results. I'm curious. Does this trigger a bug box in your system? (size_t and others are there for hysteric raisins of comparison.) with System.Storage_Elements; use System.Storage_Elements; with Ada.Unchecked_Deallocation; with Ada.Numerics.Discrete_Random, Ada.Numerics.Elementary_Functions; with Ada.Text_IO; procedure Alloc_Ada is -- powers of 2, for growing allocated sizes: Small : constant := 16; Large : constant := 40; type Void_Ptr is access Storage_Array; subtype size_t is Storage_Offset; package Size_T_IO is new Ada.Text_IO.Integer_IO (size_t); use Size_T_IO, Ada.Text_IO; procedure Init (Object : Void_Ptr; Acc : out Storage_Element); -- Initialize some of Object with random data. Place something -- almost anywhere in the allocated area, assuming that doing so -- will require commits. procedure Free is new Ada.Unchecked_Deallocation (Storage_Array, Void_Ptr); procedure Init (Object : Void_Ptr; Acc : out Storage_Element) is subtype Places is Storage_Offset range 0 .. Object'Length - 1; package Arb is new Ada.Numerics.Discrete_Random (Storage_Element); package Pos is new Ada.Numerics.Discrete_Random (Places); package Math renames Ada.Numerics.Elementary_Functions; G : Arb.Generator; P : Pos.Generator; Offset, Previous_Offset : Storage_Offset := 0; No_Of_Inits : constant Natural := 2**(Natural (Math.Log (Float (Object'Length), 2.0)) - (Small - 4)); begin Arb.Reset (G); Pos.Reset (P); Acc := 0; for Run in 1 .. No_Of_Inits loop Previous_Offset := Offset; Offset := Pos.Random (P); Object (Offset) := Arb.Random (G); Acc := Acc / 2 + (Object (Previous_Offset) + Object (Offset)) / 4; end loop; end Init; Buffer : Void_Ptr; Kept : Storage_Element; begin for Power in size_t range Small .. Large loop Put ("At "); Put (Power); Put (" ("); Put (2**Natural (Power)); Put (")"); Buffer := new Storage_Array (0 .. 2**Natural (Power)); Init (Buffer, Acc => Kept); Put_Line (Storage_Element'Image (Kept)); Free (Buffer); end loop; end Alloc_Ada; ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 11:09 Increasing GNAT's heap Dmitry A. Kazakov 2013-11-12 13:26 ` Georg Bauhaus @ 2013-11-12 22:03 ` Florian Weimer 2013-11-13 8:05 ` Dmitry A. Kazakov 2013-11-13 8:21 ` Ludovic Brenta 2013-11-14 13:07 ` Marius Amado-Alves 3 siblings, 1 reply; 29+ messages in thread From: Florian Weimer @ 2013-11-12 22:03 UTC (permalink / raw) * Dmitry A. Kazakov: > Does anybody know a way to increase the heap size available for GNAT? It > crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. > > GNAT 4.6.4 under Debian, testing, 32-bit Uhm, switch to 64 bits? ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 22:03 ` Florian Weimer @ 2013-11-13 8:05 ` Dmitry A. Kazakov 2013-11-13 15:41 ` Oliver Kleinke 0 siblings, 1 reply; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 8:05 UTC (permalink / raw) On Tue, 12 Nov 2013 23:03:36 +0100, Florian Weimer wrote: > * Dmitry A. Kazakov: > >> Does anybody know a way to increase the heap size available for GNAT? It >> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >> >> GNAT 4.6.4 under Debian, testing, 32-bit > > Uhm, switch to 64 bits? And setting up GNAT for 32-bit target? I don't know how much work that would require. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:05 ` Dmitry A. Kazakov @ 2013-11-13 15:41 ` Oliver Kleinke 0 siblings, 0 replies; 29+ messages in thread From: Oliver Kleinke @ 2013-11-13 15:41 UTC (permalink / raw) Am Wed, 13 Nov 2013 09:05:31 +0100 schrieb "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>: > On Tue, 12 Nov 2013 23:03:36 +0100, Florian Weimer wrote: > > > * Dmitry A. Kazakov: > > > >> Does anybody know a way to increase the heap size available for > >> GNAT? It crashes with famous GNAT BUG DETECTED, Storage_Error heap > >> exhausted. > >> > >> GNAT 4.6.4 under Debian, testing, 32-bit > > > > Uhm, switch to 64 bits? > > And setting up GNAT for 32-bit target? I don't know how much work that > would require. Probably negligible compared to the increased deployment effort.. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 11:09 Increasing GNAT's heap Dmitry A. Kazakov 2013-11-12 13:26 ` Georg Bauhaus 2013-11-12 22:03 ` Florian Weimer @ 2013-11-13 8:21 ` Ludovic Brenta 2013-11-13 8:23 ` Ludovic Brenta 2013-11-13 8:58 ` Dmitry A. Kazakov 2013-11-14 13:07 ` Marius Amado-Alves 3 siblings, 2 replies; 29+ messages in thread From: Ludovic Brenta @ 2013-11-13 8:21 UTC (permalink / raw) "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: > Does anybody know a way to increase the heap size available for GNAT? It > crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. > > GNAT 4.6.4 under Debian, testing, 32-bit Presuming the heap is really exhausted (i.e. the command "free" shows no free physical RAM and no free swap space left, and the command "top" shows the gnat1 process consuming close to 4 GiB of resident memory) then I'd suggest that you cannot increase the heap. So, you should compile smaller source files. I'd try to identify which subprogram causes the problem and place that subprogram in a compilation unit of its own (e.g. a private child unit). -- Ludovic Brenta. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:21 ` Ludovic Brenta @ 2013-11-13 8:23 ` Ludovic Brenta 2013-11-13 8:58 ` Dmitry A. Kazakov 1 sibling, 0 replies; 29+ messages in thread From: Ludovic Brenta @ 2013-11-13 8:23 UTC (permalink / raw) Ludovic Brenta writes on comp.lang.ada: > "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >> Does anybody know a way to increase the heap size available for GNAT? It >> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >> >> GNAT 4.6.4 under Debian, testing, 32-bit > > Presuming the heap is really exhausted (i.e. the command "free" shows no > free physical RAM and no free swap space left, and the command "top" > shows the gnat1 process consuming close to 4 GiB of resident memory) > then I'd suggest that you cannot increase the heap. So, you should > compile smaller source files. > > I'd try to identify which subprogram causes the problem and place that > subprogram in a compilation unit of its own (e.g. a private child unit). Maybe I forgot the obvious: of course, you know you can increase swap space easily by creating a file filled with zeroes and saying "swapon $the_new_file" as root. -- Ludovic Brenta. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:21 ` Ludovic Brenta 2013-11-13 8:23 ` Ludovic Brenta @ 2013-11-13 8:58 ` Dmitry A. Kazakov 2013-11-13 11:04 ` Georg Bauhaus ` (2 more replies) 1 sibling, 3 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 8:58 UTC (permalink / raw) On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote: > "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >> Does anybody know a way to increase the heap size available for GNAT? It >> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >> >> GNAT 4.6.4 under Debian, testing, 32-bit > > Presuming the heap is really exhausted (i.e. the command "free" shows no > free physical RAM and no free swap space left, and the command "top" > shows the gnat1 process consuming close to 4 GiB of resident memory) > then I'd suggest that you cannot increase the heap. So, you should > compile smaller source files. It is 8GB RAM allowed for the virtual machine. Are you sure about 4GB? Doesn't Linux reserve some address space for its the kernel? Under Windows, where GNAT has same problems I cannot get more than 2GB. There is a boot switch to allow 3GB but it does not work well with GNAT. When GNAT is running under 64-bit Windows it goes through being 32-bit itself. From this I conclude that true 4GB under Linux would be enough to compile it. > I'd try to identify which subprogram causes the problem and place that > subprogram in a compilation unit of its own (e.g. a private child unit). It is always generic instances in the case of GNAT. It appears that the GNAT compilation model is not very much separate compilation. A compiled generic package instance does not reduce the amount of memory required later at the point when another body with/use it. The compiler starts frantically allocating huge amounts of memory, expanding generics, I suppose. This is how it comes to gigabytes when compiling a mere 100-liner. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:58 ` Dmitry A. Kazakov @ 2013-11-13 11:04 ` Georg Bauhaus 2013-11-13 13:31 ` Dmitry A. Kazakov 2013-11-13 15:37 ` Oliver Kleinke 2013-11-13 15:32 ` Oliver Kleinke 2013-11-13 22:08 ` Ludovic Brenta 2 siblings, 2 replies; 29+ messages in thread From: Georg Bauhaus @ 2013-11-13 11:04 UTC (permalink / raw) On 13.11.13 09:58, Dmitry A. Kazakov wrote: > On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote: > >> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >>> Does anybody know a way to increase the heap size available for GNAT? It >>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >>> >>> GNAT 4.6.4 under Debian, testing, 32-bit > The compiler starts > frantically allocating huge amounts of memory, expanding generics, I > suppose. This is how it comes to gigabytes when compiling a mere 100-liner. Ouch, I had completely misread the question. Voracious consumption of memory is also a property of GCC for C. Gcc hits the ceiling (or floor) for even a short C program: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56344 Another way to ask, then, might be, if there is a GNAT executable that will itself use a PAE kernel's extended addressing? If doubling memory only defers the issue, maybe that's something to ask on the GCC list? ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 11:04 ` Georg Bauhaus @ 2013-11-13 13:31 ` Dmitry A. Kazakov 2013-11-13 15:39 ` Oliver Kleinke 2013-11-13 15:37 ` Oliver Kleinke 1 sibling, 1 reply; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 13:31 UTC (permalink / raw) On Wed, 13 Nov 2013 12:04:12 +0100, Georg Bauhaus wrote: > On 13.11.13 09:58, Dmitry A. Kazakov wrote: >> On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote: >> >>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >>>> Does anybody know a way to increase the heap size available for GNAT? It >>>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >>>> >>>> GNAT 4.6.4 under Debian, testing, 32-bit > >> The compiler starts >> frantically allocating huge amounts of memory, expanding generics, I >> suppose. This is how it comes to gigabytes when compiling a mere 100-liner. > > Ouch, I had completely misread the question. Voracious consumption > of memory is also a property of GCC for C. I don't believe it has anything to do with GCC. It looks clearly like GNAT issue which, BTW, also leads to extremely sluggish compilation. After all, if you allocate 2+ GB of memory you have to fill it with some junk and that takes time, much time. If AdaCore didn't fix it, that is probably because, I guess, it would require to redesign much of the compiler code. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 13:31 ` Dmitry A. Kazakov @ 2013-11-13 15:39 ` Oliver Kleinke 2013-11-13 16:07 ` Eryndlia Mavourneen 2013-11-13 17:27 ` Dmitry A. Kazakov 0 siblings, 2 replies; 29+ messages in thread From: Oliver Kleinke @ 2013-11-13 15:39 UTC (permalink / raw) > I don't believe it has anything to do with GCC. It looks clearly like > GNAT issue which, BTW, also leads to extremely sluggish compilation. > After all, if you allocate 2+ GB of memory you have to fill it with > some junk and that takes time, much time. I suppose this has to do with swapping/paging, so the sluggishness might not be GNAT's fault. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 15:39 ` Oliver Kleinke @ 2013-11-13 16:07 ` Eryndlia Mavourneen 2013-11-13 16:11 ` Eryndlia Mavourneen 2013-11-13 17:27 ` Dmitry A. Kazakov 1 sibling, 1 reply; 29+ messages in thread From: Eryndlia Mavourneen @ 2013-11-13 16:07 UTC (permalink / raw) On Wednesday, November 13, 2013 9:39:23 AM UTC-6, Oliver Kleinke wrote: > > I don't believe it has anything to do with GCC. It looks clearly like > > GNAT issue which, BTW, also leads to extremely sluggish compilation. > > After all, if you allocate 2+ GB of memory you have to fill it with > > some junk and that takes time, much time. > > I suppose this has to do with swapping/paging, so the sluggishness might > not be GNAT's fault. I was thinking along these lines. It is important to remember that the program being run dictates much of what occurs with paging. Data references aside, the locality of execution has a great deal to do with the amount of paging and swapping that occurs. A program that constantly jumps around to widely disparate parts of the address space will experience considerably more paging than a program which is as sequential as possible. I would think that it is possible to have the linker combine frequently used subprograms together with the main code in a single program section or sections that are loaded together to minimize the paging. Alternatively, there are tools which help to identify which parts of the code are being heavily executed. It then is possible to lock as many of these pages into memory as circumstances allow. This will provide a huge increase in speed with relatively little work. Not having had to worry too much about these issues in a while, I am not up on the latest tools for these things, but I am certain that others in this group are. -- Eryndlia Mavourneen (KK1T) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 16:07 ` Eryndlia Mavourneen @ 2013-11-13 16:11 ` Eryndlia Mavourneen 0 siblings, 0 replies; 29+ messages in thread From: Eryndlia Mavourneen @ 2013-11-13 16:11 UTC (permalink / raw) On Wednesday, November 13, 2013 10:07:14 AM UTC-6, Eryndlia Mavourneen wrote: > On Wednesday, November 13, 2013 9:39:23 AM UTC-6, Oliver Kleinke wrote: > > > > I don't believe it has anything to do with GCC. It looks clearly like > > > GNAT issue which, BTW, also leads to extremely sluggish compilation. > > > After all, if you allocate 2+ GB of memory you have to fill it with > > > some junk and that takes time, much time. > > > > I suppose this has to do with swapping/paging, so the sluggishness might > > not be GNAT's fault. > > I was thinking along these lines. It is important to remember that the program being run dictates much of what occurs with paging. Data references aside, the locality of execution has a great deal to do with the amount of paging and swapping that occurs. > . . . > -- Eryndlia Mavourneen (KK1T) Actually, I meant to say "paging" and not "paging and swapping," although swapping can be partially controlled by the process' priority relative to the others in the system with the same OS mode (kernel, user, etc.). -- em ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 15:39 ` Oliver Kleinke 2013-11-13 16:07 ` Eryndlia Mavourneen @ 2013-11-13 17:27 ` Dmitry A. Kazakov 2013-11-13 17:41 ` Eryndlia Mavourneen 1 sibling, 1 reply; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 17:27 UTC (permalink / raw) On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote: >> I don't believe it has anything to do with GCC. It looks clearly like >> GNAT issue which, BTW, also leads to extremely sluggish compilation. >> After all, if you allocate 2+ GB of memory you have to fill it with >> some junk and that takes time, much time. > > I suppose this has to do with swapping/paging, so the sluggishness might > not be GNAT's fault. Certainly no paging, the virtual machine has 8GB free. Compilation is also sluggish under Windows for both native and VxWorks targets on a machine with 16GB RAM. Both compile successfully. [Not to confuse compiling with linking. Linking is also extremely slow, especially when building a dynamic library, but that is a known GCC issue] -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 17:27 ` Dmitry A. Kazakov @ 2013-11-13 17:41 ` Eryndlia Mavourneen 2013-11-13 18:02 ` Dmitry A. Kazakov 0 siblings, 1 reply; 29+ messages in thread From: Eryndlia Mavourneen @ 2013-11-13 17:41 UTC (permalink / raw) On Wednesday, November 13, 2013 11:27:26 AM UTC-6, Dmitry A. Kazakov wrote: > On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote: > > >> I don't believe it has anything to do with GCC. It looks clearly like > >> GNAT issue which, BTW, also leads to extremely sluggish compilation. > >> After all, if you allocate 2+ GB of memory you have to fill it with > >> some junk and that takes time, much time. > > > > I suppose this has to do with swapping/paging, so the sluggishness might > > not be GNAT's fault. > > Certainly no paging, the virtual machine has 8GB free. > > Compilation is also sluggish under Windows for both native and VxWorks > targets on a machine with 16GB RAM. Both compile successfully. > > [Not to confuse compiling with linking. Linking is also extremely slow, > especially when building a dynamic library, but that is a known GCC issue] > > -- > Regards, > Dmitry A. Kazakov > http://www.dmitry-kazakov.de It typically is not the amount of free virtual address space (which is easy to allocate) but the amount of physical memory in addition to the references to the address space actually used by the program. These referenced pages must be read into memory (and others probably written to bulk storage to make room), when the requested addresses reside in pages that are not presently resident. -- Eryndlia Mavourneen (KK1T) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 17:41 ` Eryndlia Mavourneen @ 2013-11-13 18:02 ` Dmitry A. Kazakov 0 siblings, 0 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 18:02 UTC (permalink / raw) On Wed, 13 Nov 2013 09:41:23 -0800 (PST), Eryndlia Mavourneen wrote: > On Wednesday, November 13, 2013 11:27:26 AM UTC-6, Dmitry A. Kazakov wrote: >> On Wed, 13 Nov 2013 16:39:23 +0100, Oliver Kleinke wrote: >> >>>> I don't believe it has anything to do with GCC. It looks clearly like >>>> GNAT issue which, BTW, also leads to extremely sluggish compilation. >>>> After all, if you allocate 2+ GB of memory you have to fill it with >>>> some junk and that takes time, much time. >>> >>> I suppose this has to do with swapping/paging, so the sluggishness might >>> not be GNAT's fault. >> >> Certainly no paging, the virtual machine has 8GB free. >> >> Compilation is also sluggish under Windows for both native and VxWorks >> targets on a machine with 16GB RAM. Both compile successfully. >> >> [Not to confuse compiling with linking. Linking is also extremely slow, >> especially when building a dynamic library, but that is a known GCC issue] >> > It typically is not the amount of free virtual address space (which is > easy to allocate) but the amount of physical memory in addition to the > references to the address space actually used by the program. The guest machine has 8GB physical memory. The host machine has 16GB and shows no paging. GNAT crashes on Storage_Error, so paging/swapping is irrelevant anyway. The point was merely about that if a program, any program, uses much memory that program MUST be slow if the memory is used (= written and read). -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 11:04 ` Georg Bauhaus 2013-11-13 13:31 ` Dmitry A. Kazakov @ 2013-11-13 15:37 ` Oliver Kleinke 1 sibling, 0 replies; 29+ messages in thread From: Oliver Kleinke @ 2013-11-13 15:37 UTC (permalink / raw) > Another way to ask, then, might be, if there is a GNAT > executable that will itself use a PAE kernel's extended addressing? Each process will still be limited to the 32-bit address space, and IIRC it is not possable (I could be wrong) to have a process' memory/heap span two "PAE-pages". PS: I'm still working on your code, just need some free time :) ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:58 ` Dmitry A. Kazakov 2013-11-13 11:04 ` Georg Bauhaus @ 2013-11-13 15:32 ` Oliver Kleinke 2013-11-13 17:36 ` Dmitry A. Kazakov 2013-11-13 22:08 ` Ludovic Brenta 2 siblings, 1 reply; 29+ messages in thread From: Oliver Kleinke @ 2013-11-13 15:32 UTC (permalink / raw) > It is 8GB RAM allowed for the virtual machine. Probably a waste without PAE. > Are you sure about 4GB? Doesn't Linux reserve some address space for > its the kernel? Under Windows, where GNAT has same problems I cannot > get more than 2GB. There is a boot switch to allow 3GB but it does > not work well with GNAT. When GNAT is running under 64-bit Windows it > goes through being 32-bit itself. From this I conclude that true 4GB > under Linux would be enough to compile it. Various chunks of the 32-bit address space are used to address various hardware devices in your machine, e.g. PCI devices, APIC etc. IIRC, how much address space is "wasted" depends on the hardware. So in essence you will never get 4GiB of RAM on a 32-bit machine. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 15:32 ` Oliver Kleinke @ 2013-11-13 17:36 ` Dmitry A. Kazakov 2013-11-13 18:44 ` Oliver Kleinke 0 siblings, 1 reply; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-13 17:36 UTC (permalink / raw) On Wed, 13 Nov 2013 16:32:13 +0100, Oliver Kleinke wrote: > Various chunks of the 32-bit address space are used to address > various hardware devices in your machine, e.g. PCI devices, APIC etc. > IIRC, how much address space is "wasted" depends on the hardware. So in > essence you will never get 4GiB of RAM on a 32-bit machine. The question is whether that is closer to 4 or to 2. Under Windows it is strictly 2. If Linux reserves lesser chunks of the address space then it should be able to compile the program which at its peak takes 2.6GB for gnat1 under Windows. However, lesser chunks would mean more of them and thus more overhead for handling the process address space. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 17:36 ` Dmitry A. Kazakov @ 2013-11-13 18:44 ` Oliver Kleinke 0 siblings, 0 replies; 29+ messages in thread From: Oliver Kleinke @ 2013-11-13 18:44 UTC (permalink / raw) Am Wed, 13 Nov 2013 18:36:38 +0100 schrieb "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>: > On Wed, 13 Nov 2013 16:32:13 +0100, Oliver Kleinke wrote: > > > Various chunks of the 32-bit address space are used to address > > various hardware devices in your machine, e.g. PCI devices, APIC > > etc. IIRC, how much address space is "wasted" depends on the > > hardware. So in essence you will never get 4GiB of RAM on a 32-bit > > machine. > > The question is whether that is closer to 4 or to 2. Under Windows it > is strictly 2. If Linux reserves lesser chunks of the address space > then it should be able to compile the program which at its peak takes > 2.6GB for gnat1 under Windows. However, lesser chunks would mean more > of them and thus more overhead for handling the process address space. Should be in the range of 3 - 3.5 GiB, though it depends on your hardware (try free or cat /proc/meminfo). ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 8:58 ` Dmitry A. Kazakov 2013-11-13 11:04 ` Georg Bauhaus 2013-11-13 15:32 ` Oliver Kleinke @ 2013-11-13 22:08 ` Ludovic Brenta 2013-11-14 8:51 ` Dmitry A. Kazakov 2 siblings, 1 reply; 29+ messages in thread From: Ludovic Brenta @ 2013-11-13 22:08 UTC (permalink / raw) "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: > On Wed, 13 Nov 2013 09:21:05 +0100, Ludovic Brenta wrote: > >> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes: >>> Does anybody know a way to increase the heap size available for GNAT? It >>> crashes with famous GNAT BUG DETECTED, Storage_Error heap exhausted. >>> >>> GNAT 4.6.4 under Debian, testing, 32-bit >> >> Presuming the heap is really exhausted (i.e. the command "free" shows no >> free physical RAM and no free swap space left, and the command "top" >> shows the gnat1 process consuming close to 4 GiB of resident memory) >> then I'd suggest that you cannot increase the heap. So, you should >> compile smaller source files. > > It is 8GB RAM allowed for the virtual machine. > > Are you sure about 4GB? Doesn't Linux reserve some address space for its > the kernel? Under Windows, where GNAT has same problems I cannot get more > than 2GB. There is a boot switch to allow 3GB but it does not work well > with GNAT. When GNAT is running under 64-bit Windows it goes through being > 32-bit itself. From this I conclude that true 4GB under Linux would be > enough to compile it. You did mention "GNAT 4.6.4 under Debian, testing, 32-bit". Now the actual amount of virtual address space available to a 32-bit process under Linux depends on the Linux kernel: pure 32-bit kernel => 2 or 3 GiB, depending on configuration 32-bit kernel with PAE => 4 GiB per process 64-bit kernel => 4 GiB per 32-bit process >> I'd try to identify which subprogram causes the problem and place >> that subprogram in a compilation unit of its own (e.g. a private >> child unit). > > It is always generic instances in the case of GNAT. It appears that > the GNAT compilation model is not very much separate compilation. A > compiled generic package instance does not reduce the amount of memory > required later at the point when another body with/use it. The > compiler starts frantically allocating huge amounts of memory, > expanding generics, I suppose. This is how it comes to gigabytes when > compiling a mere 100-liner. Then maybe you should try placing the instantiation of your generic into a package of its own, thereby breaking a chain of instantiations? An ugly workaround may be to declare a package with subprograms, all of which rename the subprograms defined by a generic instantiation that is hidden in the body, e.g. generic type Element (<>) is private; package Huge_Generic_G is -- consumes lots of RAM in the compiler procedure A (E : in out Element); end Huge_Generic_G; -- body omitted package Huge_Generic_Instantiation -- compiled separately type Element is record ... end record; procedure A (E : in out Element); end Huge_Generic_Instantiation; with Huge_Generic_G; package body Huge_Generic_Instantiation is The_Instantiation_That_Consumes_Lots_Of_Memory is new Huge_Generic_G (Element => Element); procedure A (E : in out Element) renames The_Instantiation_That_Consumes_Lots_Of_Memory.A; end Huge_Generic_G; ? -- Ludovic Brenta. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-13 22:08 ` Ludovic Brenta @ 2013-11-14 8:51 ` Dmitry A. Kazakov 2013-11-14 10:50 ` Georg Bauhaus 0 siblings, 1 reply; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-14 8:51 UTC (permalink / raw) On Wed, 13 Nov 2013 23:08:56 +0100, Ludovic Brenta wrote: > You did mention "GNAT 4.6.4 under Debian, testing, 32-bit". Now the > actual amount of virtual address space available to a 32-bit process > under Linux depends on the Linux kernel: > > pure 32-bit kernel => 2 or 3 GiB, depending on configuration > 32-bit kernel with PAE => 4 GiB per process > 64-bit kernel => 4 GiB per 32-bit process linux-image-3.10-3-686-pae, but it seems that GNAT cannot get more than 2GB or so. > Then maybe you should try placing the instantiation of your generic into > a package of its own, thereby breaking a chain of instantiations? I cannot instantiate in a package, because instances come from a library. They are a part of another project. Trying to restructure the program, it seems that if a package A with-es B that inturn does C, then compilation of A will still be influenced by C even if A does not refer to C. Separate bodies do not help either. It looks like if a separate body with-es anything that swings back to the parent body. -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-14 8:51 ` Dmitry A. Kazakov @ 2013-11-14 10:50 ` Georg Bauhaus 2013-11-14 11:16 ` Dmitry A. Kazakov 0 siblings, 1 reply; 29+ messages in thread From: Georg Bauhaus @ 2013-11-14 10:50 UTC (permalink / raw) On 14.11.13 09:51, Dmitry A. Kazakov wrote: > Trying to restructure the program, it seems that if a package A with-es B > that inturn does C, then compilation of A will still be influenced by C > even if A does not refer to C. Just a wild guess: if there is some huge elaboration going on in the generic units, does -gnatE have an effect? ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-14 10:50 ` Georg Bauhaus @ 2013-11-14 11:16 ` Dmitry A. Kazakov 0 siblings, 0 replies; 29+ messages in thread From: Dmitry A. Kazakov @ 2013-11-14 11:16 UTC (permalink / raw) On Thu, 14 Nov 2013 11:50:00 +0100, Georg Bauhaus wrote: > On 14.11.13 09:51, Dmitry A. Kazakov wrote: >> Trying to restructure the program, it seems that if a package A with-es B >> that inturn does C, then compilation of A will still be influenced by C >> even if A does not refer to C. > > Just a wild guess: if there is some huge elaboration > going on in the generic units, does -gnatE have an effect? No -- Regards, Dmitry A. Kazakov http://www.dmitry-kazakov.de ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: Increasing GNAT's heap 2013-11-12 11:09 Increasing GNAT's heap Dmitry A. Kazakov ` (2 preceding siblings ...) 2013-11-13 8:21 ` Ludovic Brenta @ 2013-11-14 13:07 ` Marius Amado-Alves 3 siblings, 0 replies; 29+ messages in thread From: Marius Amado-Alves @ 2013-11-14 13:07 UTC (permalink / raw) When I need a lot of memory I create a task and define its memory size with the pragma. I recall defining a size of 200_000_000 bytes on laptops and desktops with GNAT on Windows and Macs. I recall having to create large objects viz. matrices *dynamically*. I recall dabbling with ulimit but I don't recall if it was necessary. ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2013-11-14 13:07 UTC | newest] Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-11-12 11:09 Increasing GNAT's heap Dmitry A. Kazakov 2013-11-12 13:26 ` Georg Bauhaus 2013-11-12 14:00 ` Dmitry A. Kazakov 2013-11-12 17:31 ` Paul Rubin 2013-11-13 8:00 ` Dmitry A. Kazakov 2013-11-12 19:30 ` Georg Bauhaus 2013-11-12 22:03 ` Florian Weimer 2013-11-13 8:05 ` Dmitry A. Kazakov 2013-11-13 15:41 ` Oliver Kleinke 2013-11-13 8:21 ` Ludovic Brenta 2013-11-13 8:23 ` Ludovic Brenta 2013-11-13 8:58 ` Dmitry A. Kazakov 2013-11-13 11:04 ` Georg Bauhaus 2013-11-13 13:31 ` Dmitry A. Kazakov 2013-11-13 15:39 ` Oliver Kleinke 2013-11-13 16:07 ` Eryndlia Mavourneen 2013-11-13 16:11 ` Eryndlia Mavourneen 2013-11-13 17:27 ` Dmitry A. Kazakov 2013-11-13 17:41 ` Eryndlia Mavourneen 2013-11-13 18:02 ` Dmitry A. Kazakov 2013-11-13 15:37 ` Oliver Kleinke 2013-11-13 15:32 ` Oliver Kleinke 2013-11-13 17:36 ` Dmitry A. Kazakov 2013-11-13 18:44 ` Oliver Kleinke 2013-11-13 22:08 ` Ludovic Brenta 2013-11-14 8:51 ` Dmitry A. Kazakov 2013-11-14 10:50 ` Georg Bauhaus 2013-11-14 11:16 ` Dmitry A. Kazakov 2013-11-14 13:07 ` Marius Amado-Alves
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox