many exceptions cause memory leak?

many exceptions cause memory leak?

[Christopher Broeg]

Hi,

I am running some numeric code in ADA95 using GNAT Academic edition. 
When negative values occur or the integration routine fails exceptions 
are raised and propagated. This works fine.

Surprisingly, the memory usage increases from an initial ~40MB to 100MB 
and more during one day of execution. the memory usage shouldn't 
increase in my eyes. Is it possible that raising many exceptions 
coninues to use up some memory?

thanks for your ideas,

Chris

Re: many exceptions cause memory leak?

[Frank J. Lhota]

Christopher Broeg wrote:
> Hi,
> 
> I am running some numeric code in ADA95 using GNAT Academic edition. 
> When negative values occur or the integration routine fails exceptions 
> are raised and propagated. This works fine.
> 
> Surprisingly, the memory usage increases from an initial ~40MB to 100MB 
> and more during one day of execution. the memory usage shouldn't 
> increase in my eyes. Is it possible that raising many exceptions 
> coninues to use up some memory?
> 
> thanks for your ideas,
> 
> Chris

Which platform is this running on? That could be helpful in determining 
the cause of your problem.

On Windows XP, I wrote a GNAT GPL application that raised and handled a 
million exception occurrences. This application also does a call to the 
Win32 API function GlobalMemoryStatus both before and after handling a 
million exception occurrences. The memory status before and after 
handling these exceptions was virtually the same. This seems to indicate 
that on XP at least, GNAT exceptions do *not* leak memory.

-- 
"All things extant in this world,
Gods of Heaven, gods of Earth,
Let everything be as it should be;
Thus shall it be!"
- Magical chant from "Magical Shopping Arcade Abenobashi"

"Drizzle, Drazzle, Drozzle, Drome,
Time for the this one to come home!"
- Mr. Lizard from "Tutor Turtle"

Re: many exceptions cause memory leak?

[Robert Klungle]

For what it is worth, a few years ago my team developed a large system which
ran on SGIs, IBM AIXs, and Intel systems. We found that over time, the
memory usage by the system grew even though all memory was being returned.
We found out that the Ada Run Time requested more memory from the OS even
though it had memory available in its allocation pool. Eventually, this pool
allocation stopped growing and occasionally dropped a little (we expect due
to the action of other programs running). The Ada allocation pool grew to
4.6 GB before stabilizing. The system ran for weeks without crashing (due to
memory exhaustion). We were told by the compiler vendor this was done to
improve the dynamic memory allocation performance. Take it for what it is
worth.

Just an observation.

cheers...bob

"Frank J. Lhota" <NOSPAM.lhota@adarose.com> wrote in message
news:yX76f.7$dW6.4@trndny09...
> Christopher Broeg wrote:
> > Hi,
> >
> > I am running some numeric code in ADA95 using GNAT Academic edition.
> > When negative values occur or the integration routine fails exceptions
> > are raised and propagated. This works fine.
> >
> > Surprisingly, the memory usage increases from an initial ~40MB to 100MB
> > and more during one day of execution. the memory usage shouldn't
> > increase in my eyes. Is it possible that raising many exceptions
> > coninues to use up some memory?
> >
> > thanks for your ideas,
> >
> > Chris
>
> Which platform is this running on? That could be helpful in determining
> the cause of your problem.
>
> On Windows XP, I wrote a GNAT GPL application that raised and handled a
> million exception occurrences. This application also does a call to the
> Win32 API function GlobalMemoryStatus both before and after handling a
> million exception occurrences. The memory status before and after
> handling these exceptions was virtually the same. This seems to indicate
> that on XP at least, GNAT exceptions do *not* leak memory.
>
> -- 
> "All things extant in this world,
> Gods of Heaven, gods of Earth,
> Let everything be as it should be;
> Thus shall it be!"
> - Magical chant from "Magical Shopping Arcade Abenobashi"
>
> "Drizzle, Drazzle, Drozzle, Drome,
> Time for the this one to come home!"
> - Mr. Lizard from "Tutor Turtle"

Re: many exceptions cause memory leak?

[Ed Falis]

On Fri, 21 Oct 2005 13:59:36 -0400, Robert Klungle <bklungle@adelphia.net>  
wrote:

>  We were told by the compiler vendor this was done to
> improve the dynamic memory allocation performance. Take it for what it is
> worth.

Sounds like an old Alsys compiler that grabbed large chunks of memory,  
then managed the allocated chunks as the application's heap.  This as  
opposed to using malloc directly.

- Ed

Re: many exceptions cause memory leak?

[Christopher Broeg]

Hi,

sorry that I didn't reply immediately. I was away over the weekend.

 From your replys I assume that ada should not waste memory by exception 
calls. Thanks for that.

Well, my platform is suse Linux and I compile using
GNATMAKE GAP 1.1.0 (20041209-323)

I am by no means an expert in programming but I figured that all memory 
assigned by pointers is returned. I do not use my own storage pool but 
allocate just by
is new...
so it is all on the heap, I guess. The application is not large on your 
terms. It is just an a little more complicated numerical algorithm 
finding a few thousand solutions to a set of ordinary differential 
equations. Using spline interpolation for equations of state, and other 
physical quantities, it gets a little more complicated, though. 
Furthermore my data i/o is via a c binding to HDF5. I am not sure if may 
be the HDF5 library wasting the memory.

It is not real problem, because the program usually finishes around a 
few 100 MB, but when I run the calculations on other peoples machines I 
don't want it to use so much (it is nice of them to allow the cpu load, 
already). Upon starting it only uses 46 MB. But increases steadily.

Any suggestions how to find the memory leak?

Thanks,

Chris

Re: many exceptions cause memory leak?

[Simon Wright]

Christopher Broeg <broeg@astro.uni-jena.de> writes:

> I am by no means an expert in programming but I figured that all
> memory assigned by pointers is returned. I do not use my own storage
> pool but allocate just by
> is new...
> so it is all on the heap, I guess.

GNAT doesn't automatically free this memory until the end of the
program. You need to free it yourself.

Re: many exceptions cause memory leak?

[Christopher Broeg]

Simon Wright wrote:
> Christopher Broeg <broeg@astro.uni-jena.de> writes:
> 
>>I am by no means an expert in programming but I figured that all
>>memory assigned by pointers is returned. I do not use my own storage
>>pool but allocate just by
>>is new...
>>so it is all on the heap, I guess.
> 
> GNAT doesn't automatically free this memory until the end of the
> program. You need to free it yourself.

I do use unchecked deallocation on all pointers. I now have used gnatmem 
and found one memory leak that was my fault. There are still a few 
thousand "leaks" due to ada exception handling, though. Gnatmem output 
follows:

Global information
------------------
    Total number of allocations        :109993
    Total number of deallocations      :97243
    Final Water Mark (non freed mem)   :  29.85 Megabytes
    High Water Mark                    :  29.90 Megabytes

Allocation Root # 1
-------------------
  Number of non freed allocations    :12702
  Final Water Mark (non freed mem)   :  80.87 Kilobytes
  High Water Mark                    :  80.87 Kilobytes
  Backtrace                          :
    a-except.adb:1396 <ada__exceptions___elabb>

I do think that the program now stays constnat in size, or at least 
asymtotically approaches a relatively small constant size in memory. 
running the code over night, the high water mark of the entire program 
was small, so I hope the problem is solved. But since the program had 
already finished by this morning, I couldn't check independently using 
top, also I could not check whether this means reserved memory only and 
what happens with the virtual memory. It used to be twice as large as 
the reserved. I'll know in another 10 hours if everything is fine now.

Still, I am wondering if the above begaviour is normal. It is only 60kB, 
so no big deal, but 12702 non freed allocations? Is this normal? They 
always stem from line 1396 in a-except.adb. It is the place where a 
dummy procecdure called ZZZ is called. It appears to me that this should 
be a one-time per program lifetime event. Does this mean that there are 
12702 somehow independent exception occurences in my program?

thanks for all your help,

Chris

Re: many exceptions cause memory leak?

[Christopher Broeg]

Christopher Broeg wrote:
> Simon Wright wrote:
>> Christopher Broeg <broeg@astro.uni-jena.de> writes:
>>
>>> I am by no means an expert in programming but I figured that all
>>> memory assigned by pointers is returned. I do not use my own storage
>>> pool but allocate just by
>>> is new...
>>> so it is all on the heap, I guess.
>>
>> GNAT doesn't automatically free this memory until the end of the
>> program. You need to free it yourself.
> 
> I do use unchecked deallocation on all pointers. I now have used gnatmem 
> and found one memory leak that was my fault. There are still a few 
> thousand "leaks" due to ada exception handling, though. Gnatmem output 
> follows:
> 
> Global information
> ------------------
>    Total number of allocations        :109993
>    Total number of deallocations      :97243
>    Final Water Mark (non freed mem)   :  29.85 Megabytes
>    High Water Mark                    :  29.90 Megabytes
> 
> Allocation Root # 1
> -------------------
>  Number of non freed allocations    :12702
>  Final Water Mark (non freed mem)   :  80.87 Kilobytes
>  High Water Mark                    :  80.87 Kilobytes
>  Backtrace                          :
>    a-except.adb:1396 <ada__exceptions___elabb>
> 
> I do think that the program now stays constnat in size, or at least 
> asymtotically approaches a relatively small constant size in memory. 
> running the code over night, the high water mark of the entire program 
> was small, so I hope the problem is solved. But since the program had 
> already finished by this morning, I couldn't check independently using 
> top, also I could not check whether this means reserved memory only and 
> what happens with the virtual memory. It used to be twice as large as 
> the reserved. I'll know in another 10 hours if everything is fine now.
> 
> Still, I am wondering if the above begaviour is normal. It is only 60kB, 
> so no big deal, but 12702 non freed allocations? Is this normal? They 
> always stem from line 1396 in a-except.adb. It is the place where a 
> dummy procecdure called ZZZ is called. It appears to me that this should 
> be a one-time per program lifetime event. Does this mean that there are 
> 12702 somehow independent exception occurences in my program?
> 
> thanks for all your help,
> 
> Chris
Addition:
Could compiling with debug information be the reason for the above 
behaviour?

Chris

Re: many exceptions cause memory leak?

[Alex R. Mosteo]

Christopher Broeg wrote:

> Addition:
> Could compiling with debug information be the reason for the above 
> behaviour?

Not in my experience.

> Chris

Re: many exceptions cause memory leak?

[Alex R. Mosteo]

Alex R. Mosteo wrote:
> Christopher Broeg wrote:
> 
>> Addition:
>> Could compiling with debug information be the reason for the above 
>> behaviour?
> 
> 
> Not in my experience.

Unless you're using the Gnat.Debug_Pools package. These pools store a 
list with accessed addresses or something like that, which grows with time.

> 
>> Chris

Re: many exceptions cause memory leak?

[Wiljan Derks]

"Christopher Broeg" <broeg@astro.uni-jena.de> wrote in message 
news:djknki$cl6$1@lc03.rz.uni-jena.de...
>> Global information
>> ------------------
>>    Total number of allocations        :109993
>>    Total number of deallocations      :97243
>>    Final Water Mark (non freed mem)   :  29.85 Megabytes
>>    High Water Mark                    :  29.90 Megabytes
Still looks like you are loosing memory.
Maybe a good tip:
When and exception occurs one can easily asume that an inout variable of a 
procedure
alsready has the value assigned to its end location. Thus something like 
this:
procedure zzz(x: in out access y) is
begin
    x:=new ...
   exception here
end;
When an exception occurs, one could assume that the new pointer is already 
assigned
to the structure that was passed to zzz.
That is however not the case when using gnat.
Typically, the inout parameter is a local variable for the procedure that is 
stored on its final location when doing
a normal procedure exit.
This is a typical reason for loosing memory when doing exception processing.
Changing the mode from inout to and access parameter fixes that.

Re: many exceptions cause memory leak?

[Christopher Broeg]

Wiljan Derks wrote:
> "Christopher Broeg" <broeg@astro.uni-jena.de> wrote in message 
> news:djknki$cl6$1@lc03.rz.uni-jena.de...
>>>Global information
>>>------------------
>>>   Total number of allocations        :109993
>>>   Total number of deallocations      :97243
>>>   Final Water Mark (non freed mem)   :  29.85 Megabytes
>>>   High Water Mark                    :  29.90 Megabytes
> Still looks like you are loosing memory.
> Maybe a good tip:
> When and exception occurs one can easily asume that an inout variable of a 
> procedure
> alsready has the value assigned to its end location. Thus something like 
> this:
> procedure zzz(x: in out access y) is
> begin
>     x:=new ...
>    exception here
> end;
> When an exception occurs, one could assume that the new pointer is already 
> assigned
> to the structure that was passed to zzz.
> That is however not the case when using gnat.
> Typically, the inout parameter is a local variable for the procedure that is 
> stored on its final location when doing
> a normal procedure exit.
> This is a typical reason for loosing memory when doing exception processing.
> Changing the mode from inout to and access parameter fixes that.
> 
> 
Hi,

the ZZZ procedure is a procedure in the GNAT exception library. There 
are no pointers there, actually.

I think I might have found the reason for the memory loss, though:

I am using unchecked_acces to the data for the external HDF5 calls. The 
memory leak seems to be originating there.

If I have aliased variables do I have to manually free the memory?

i.e. I have:

declare
  Data      : H5_double.Data_1d(0..2) ;
begin
...
Status := H5_Double.H5aread (Attr_Id     => Attribute,
                              Mem_Type_Id => H5t_Native_double,
                              Buf         => Data(0)'Unchecked_Access);



data_1d is an array of aliased float. I don't do any manual deallocaton.

cheers,

Chris

Re: many exceptions cause memory leak?

[Frank J. Lhota]

Do you ever call the following function from the Ada.Exceptions package?

    function Save_Occurrence
      (Source : Exception_Occurrence)
       return   Exception_Occurrence_Access;

This function allocates an Exception Occurrence from the heap and copies 
the data from Source into this new occurrence. It is up to the 
programmer to deallocate the result of this function. An instance of 
Unchecked_Conversion can be used for this purpose.

Failure to deallocate the results of the Save_Occurrence function would 
certainly result in a memory leak.

Also, is there any other Ada.Exceptions subprogram that you call? More 
info on how you are using exceptions might help us diagnose the problem.

-- 
"All things extant in this world,
Gods of Heaven, gods of Earth,
Let everything be as it should be;
Thus shall it be!"
- Magical chant from "Magical Shopping Arcade Abenobashi"

"Drizzle, Drazzle, Drozzle, Drome,
Time for the this one to come home!"
- Mr. Lizard from "Tutor Turtle"

Re: many exceptions cause memory leak?

[Christopher Broeg]

Frank J. Lhota wrote:
> Do you ever call the following function from the Ada.Exceptions package?
> 
>    function Save_Occurrence
>      (Source : Exception_Occurrence)
>       return   Exception_Occurrence_Access;
> 
> This function allocates an Exception Occurrence from the heap and copies 
> the data from Source into this new occurrence. It is up to the 
> programmer to deallocate the result of this function. An instance of 
> Unchecked_Conversion can be used for this purpose.
> 
> Failure to deallocate the results of the Save_Occurrence function would 
> certainly result in a memory leak.
> 
> Also, is there any other Ada.Exceptions subprogram that you call? More 
> info on how you are using exceptions might help us diagnose the problem.
> 

Hi,

no I do not use save_occurence. I do use Raise_Exception and 
exception_information / message, though.

I am still running the code and it seems that the major memory leak 
problem is solved:

Global information
------------------
    Total number of allocations        :4216666
    Total number of deallocations      :3823682
    Final Water Mark (non freed mem)   :  32.21 Megabytes
    High Water Mark                    :  32.26 Megabytes

However, it appears strange to me that gnatmem detects so many non freed 
allocations originating in the exception package:

Allocation Root # 1
-------------------
  Number of non freed allocations    :392937
  Final Water Mark (non freed mem)   :   2.44 Megabytes
  High Water Mark                    :   2.44 Megabytes
  Backtrace                          :
    a-except.adb:1396 <ada__exceptions___elabb>

  Of course, the program is still running, but in the past those 
occurences have not been freed upon termination. The memory use of 2.44 
MB fits well with the fact that the code started out with a high water 
mark of roughly 29 MB and during the day raised the value to 32.21 MB. 
To be clear, this poses no problem at all to me, now. I am just curious 
and would like to understand what is going on.

I thank you all for the helpful comments and suggestions.

Chris

Re: many exceptions cause memory leak?

[Andreas Schulz]

Christopher Broeg wrote:

> Any suggestions how to find the memory leak?

gnatmem should be able to tell you - 

<http://gcc.gnu.org/onlinedocs/gcc-4.0.2/gnat_ugn_unw/The-gnatmem-Tool.html>

-- 
For whatever-the NOSPAM address is valid and mine...

Re: many exceptions cause memory leak?

[Christopher Broeg]

Hi everybody,

I finally found the (second) reason for the memory leak:

I was calling HDF5 library (c version) functions. When a string was 
needed I passed it using

name => cs.new_string("whatever")

this created a new c-pointer and i didn't deallocate it. Stupid mistake, 
really. It is the first time I call foreign libraries, so I didn't 
comprehend fully what the new_string does.

Thank you all very much for your help!


chris