Is the Ada run-time required to detect out-of-memory conditions?

Is the Ada run-time required to detect out-of-memory conditions?

[Ludovic Brenta]

Hello,

I have received a bug report [1] against GNAT 3.15p and reproduced the
same behaviour with GCC 3.4.  I would like the advice of language
lawyers on the issue.

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076

The program allocates memory in an infinite loop using "new" and the
default storage pool, and of course exhausts all available storage.
The issue is whether or not Storage_Error should be raised.  With
GNAT, no exception is raised; instead the program receives SIGKILL
(not even SIGSEGV) from Linux.

My first impression is that the program exhausts physical memory (RAM
+ swap) before exhausting the default storage pool, the size of which
is defined to be the total size of the virtual address space.  As a
result, the kernel detects the out-of-memory condition before the Ada
run-time library does, and sends SIGKILL to the program.  Of course,
the program has no chance to turn SIGKILL into an exception.

Is my assessment correct?  Is this a bug in the GNAT library?

Thanks for any insight.

PS. The original reported says that the program receives SIGSEGV.  He
uses Linux 2.4.22.  I use 2.4.26 and the program receives SIGKILL
instead.  I don't know if this is a known change in the kernel's
behaviour.

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Georg Bauhaus]

Ludovic Brenta <ludovic.brenta@insalien.org> wrote:
: PS. The original reported says that the program receives SIGSEGV.  He
: uses Linux 2.4.22.  I use 2.4.26 and the program receives SIGKILL
: instead.  I don't know if this is a known change in the kernel's
: behaviour.

also Killed when OS is 2.4.24, Debian stable.


-- Georg

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Stephen Leake]

Ludovic Brenta <ludovic.brenta@insalien.org> writes:

> Hello,
> 
> I have received a bug report [1] against GNAT 3.15p and reproduced the
> same behaviour with GCC 3.4.  I would like the advice of language
> lawyers on the issue.
> 
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
> 
> The program allocates memory in an infinite loop using "new" and the
> default storage pool, and of course exhausts all available storage.
> The issue is whether or not Storage_Error should be raised.  With
> GNAT, no exception is raised; instead the program receives SIGKILL
> (not even SIGSEGV) from Linux.

As the GNAT manual says, to get Storage_Error here, you have to
compile with -fstack-check. This is an expensive check, so it is off
by default.

-- 
-- Stephe

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Georg Bauhaus]

Stephen Leake <stephen_leake@acm.org> wrote:
: As the GNAT manual says, to get Storage_Error here, you have to
: compile with -fstack-check. This is an expensive check, so it is off
: by default.

georg@sonnenregen:/tmp$ gnatmake -g -fstack-check -s memory.adb 
gcc -c -g -fstack-check memory.adb
gnatbind -x memory.ali
gnatlink -g memory.ali
georg@sonnenregen:/tmp$ ./memory 
Killed


(Is the stack involved at all?)

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Stephen Leake]

Georg Bauhaus <sb463ba@l1-hrz.uni-duisburg.de> writes:

> Stephen Leake <stephen_leake@acm.org> wrote:
> : As the GNAT manual says, to get Storage_Error here, you have to
> : compile with -fstack-check. This is an expensive check, so it is off
> : by default.
> 
> georg@sonnenregen:/tmp$ gnatmake -g -fstack-check -s memory.adb 
> gcc -c -g -fstack-check memory.adb
> gnatbind -x memory.ali
> gnatlink -g memory.ali
> georg@sonnenregen:/tmp$ ./memory 
> Killed
> 
> 
> (Is the stack involved at all?)

Hmm. Good point. There's probably another compiler option to set. I
recently ran across -fstack-check, so I assumed it was that :).

-- 
-- Stephe

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Robert I. Eachus]

Ludovic Brenta wrote:

> I have received a bug report [1] against GNAT 3.15p and reproduced the
> same behaviour with GCC 3.4.  I would like the advice of language
> lawyers on the issue.
> 
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
> 
> The program allocates memory in an infinite loop using "new" and the
> default storage pool, and of course exhausts all available storage.
> The issue is whether or not Storage_Error should be raised.  With
> GNAT, no exception is raised; instead the program receives SIGKILL
> (not even SIGSEGV) from Linux.
> 
> My first impression is that the program exhausts physical memory (RAM
> + swap) before exhausting the default storage pool, the size of which
> is defined to be the total size of the virtual address space.  As a
> result, the kernel detects the out-of-memory condition before the Ada
> run-time library does, and sends SIGKILL to the program.  Of course,
> the program has no chance to turn SIGKILL into an exception.
> 
> Is my assessment correct?  Is this a bug in the GNAT library?
> 
> Thanks for any insight.
> 
> PS. The original reported says that the program receives SIGSEGV.  He
> uses Linux 2.4.22.  I use 2.4.26 and the program receives SIGKILL
> instead.  I don't know if this is a known change in the kernel's
> behaviour.

Unfortunately you are going to have to determine this for yourself.  Let 
me suggest a program and you can determine what should happen.  But the 
bug as such is not a GNAT library bug, it is an interaction between GNAT 
and Debian.

Write a program that exhausts memory but prints the amount of memory 
allocated and the bounds with each allocation.  You can be smart about 
this, but eventually you may have to have an allocate, write, delay loop 
in the critical area.

My guess is that you will find that what happens is that Debian 
allocates pages but only populates them on write.  So the amount of 
memory allocated eventually exceeds 2**31, even if you don't have 2 Gig 
of paging space.  Now all of a sudden you have a segment of memory 
allocated across the 2 Gig point.  And somewhere along the line some 
code is computing the size of the space allocated by address(Last) - 
address(First).  If this is in fact the case, yes there is a bug, but I 
don't know where.

If you can catch it in the debugger great.  But it sounds like you are 
going to have to figure out from the chunk that is getting allocated 
when the bomb goes off what broke.

Dave Emery used to call Storage_Error a parachute that opens on impact, 
and this is part of why.  Often there is nothing sensible that you can 
do in an local Ada exception handler, you have to try to catch it 
partway down the call stack, if possible.  But usually you are lucky if 
you can catch it inside the program.

-- 

                                           Robert I. Eachus

"The terrorists rejoice in the killing of the innocent, and have 
promised similar violence against Americans, against all free peoples, 
and against any Muslims who reject their ideology of murder. Their 
barbarism cannot be appeased, and their hatred cannot be satisfied. 
There's only one way to deal with terror: We must confront the enemy and 
stay on the offensive until these killers are defeated." -- George W. Bush

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Vinzent 'Gadget' Hoefler]

Ludovic Brenta wrote:

>[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
>
>The program allocates memory in an infinite loop using "new" and the
>default storage pool, and of course exhausts all available storage.
>The issue is whether or not Storage_Error should be raised.  With
>GNAT, no exception is raised; instead the program receives SIGKILL
>(not even SIGSEGV) from Linux.

Well, as long as these darn modern operating systems let you use
memory that isn't even there and then just kill the process instead of
signaling an error, there's no chance that the runtime can raise
Storage_Error.

Same with malloc(). Chances are much higher you just get killed than
this one ever returns NULL that you can handle. And everyone still
keeps telling you, you should check the return value. ;-(


Vinzent.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Martin Krischik]

Vinzent 'Gadget' Hoefler wrote:

> Ludovic Brenta wrote:
> 
>>[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
>>
>>The program allocates memory in an infinite loop using "new" and the
>>default storage pool, and of course exhausts all available storage.
>>The issue is whether or not Storage_Error should be raised.  With
>>GNAT, no exception is raised; instead the program receives SIGKILL
>>(not even SIGSEGV) from Linux.
> 
> Well, as long as these darn modern operating systems let you use
> memory that isn't even there and then just kill the process instead of
> signaling an error, there's no chance that the runtime can raise
> Storage_Error.

Actually, with memory in modern computer in the GB and harddrives in the 100
GB range the program is more likely to exhaust its user address space
before the available swapper fills up.

But the result is the same: No Storage_Error.

With Regards

Martin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Wojtek Narczynski]

Hello,

> I have received a bug report [1] against GNAT 3.15p and reproduced the
> same behaviour with GCC 3.4.  I would like the advice of language
> lawyers on the issue.
> 
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
> 
> The program allocates memory in an infinite loop using "new" and the
> default storage pool, and of course exhausts all available storage.
> The issue is whether or not Storage_Error should be raised.  With
> GNAT, no exception is raised; instead the program receives SIGKILL
> (not even SIGSEGV) from Linux.

I noticed, that if the program allocates memory in 6Kb chunks or
larger, Storate_Error is raised. If the chunk size is 5Kb or less, the
program only prints "Killed". In between I have not tested.

Also, shouldn't cases like this one, as a rule, eventually become part
of ACATS?

Regards,
Wojtek

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Martin Krischik]

Wojtek Narczynski wrote:

> Hello,
> 
>> I have received a bug report [1] against GNAT 3.15p and reproduced the
>> same behaviour with GCC 3.4.  I would like the advice of language
>> lawyers on the issue.
>> 
>> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
>> 
>> The program allocates memory in an infinite loop using "new" and the
>> default storage pool, and of course exhausts all available storage.
>> The issue is whether or not Storage_Error should be raised.  With
>> GNAT, no exception is raised; instead the program receives SIGKILL
>> (not even SIGSEGV) from Linux.
> 
> I noticed, that if the program allocates memory in 6Kb chunks or
> larger, Storate_Error is raised. If the chunk size is 5Kb or less, the
> program only prints "Killed". In between I have not tested.
> 
> Also, shouldn't cases like this one, as a rule, eventually become part
> of ACATS?

Just a thought: The page size on Intel CPU's is 4 kb. So most OS's on Intel
CPU will only handle memory requests of at least one page.

I find it interesting that with request significantly larger then one page
everything seems Ok.

Lets Think: With GNAT allocation from the standart storrage pool are just
passed down to C's malloc ().

When smaller memory is requested most C compiler will bundele several
requests into one page request.

When larger  memory is requested some C compiler will just pass the request
on to the OS and some other will allways bundle.

With Regards

Martin

-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Robert I. Eachus]

Martin Krischik wrote:

> Just a thought: The page size on Intel CPU's is 4 kb. So most OS's on Intel
> CPU will only handle memory requests of at least one page.
> 
> I find it interesting that with request significantly larger then one page
> everything seems Ok.
> 
> Lets Think: With GNAT allocation from the standart storrage pool are just
> passed down to C's malloc ().
> 
> When smaller memory is requested most C compiler will bundele several
> requests into one page request.
> 
> When larger  memory is requested some C compiler will just pass the request
> on to the OS and some other will allways bundle.

This is interesting.  It sounds like almost the opposite of what I 
expected.  Apparently if the memory requests are large enough that some 
pages are left virtual, you get the Storage_Error.  But if the code 
results in every page allocated being 'touched' you get killed.

But this discovery definitely indicates it is some code in malloc that 
is skipping a necessary check, and that the error is not in the GNAT 
runtime.

-- 

                                           Robert I. Eachus

"The terrorists rejoice in the killing of the innocent, and have 
promised similar violence against Americans, against all free peoples, 
and against any Muslims who reject their ideology of murder. Their 
barbarism cannot be appeased, and their hatred cannot be satisfied. 
There's only one way to deal with terror: We must confront the enemy and 
stay on the offensive until these killers are defeated." -- George W. Bush

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Mark H Johnson]

Ludovic Brenta wrote:
> Hello,
> 
> I have received a bug report [1] against GNAT 3.15p and reproduced the
> same behaviour with GCC 3.4.  [snip]
> My first impression is that the program exhausts physical memory (RAM
> + swap) before exhausting the default storage pool, the size of which
> is defined to be the total size of the virtual address space.  As a
> result, the kernel detects the out-of-memory condition before the Ada
> run-time library does, and sends SIGKILL to the program.  Of course,
> the program has no chance to turn SIGKILL into an exception.
> 
> Is my assessment correct?  Is this a bug in the GNAT library?
Well - if the OOM killer is getting rid of the application, you should 
be able to do something like the following...
  - allocate a bunch of storage with program A and then wait
  - run another copy of A and then wait
repeat running new copies until someone gets killed. If its the OOM 
killer and it kills any but the last copy - it cannot be a GNAT library 
problem. You HAD enough storage when you went into the wait but globally 
don't now. I don't see how any run time could prevent such a situation 
(especially since the program that "broke the camel's back" could be 
implemented in any language).

Also - have you checked the console / system logs for any indication 
that the OOM killer is at work?
   --Mark

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Ludovic Brenta]

Here are some more data points that seem to confirm my initial
assessment.  The process was killed by the kernel when I had 256 MiB
of RAM and 256 MiB of swap.  I created additional swap files so I now
have 256 MiB RAM and 4 GiB swap.  I now get a Storage_Error after
approximately 192 million iterations.
Ada.Exceptions.Exception_Information outputs:

Exception name: STORAGE_ERROR
Message: heap exhausted

Here is the backtrace at the point where the exception is raised:

(gdb) bt
#0  0x40091950 in __gnat_raise_nodefer_with_msg ()
   from /usr/lib/libgnat-3.15p.so.1
#1  0x40091d18 in __gnat_raise_with_msg () from /usr/lib/libgnat-3.15p.so.1
#2  0x40091ab2 in __gnat_raise_exception () from /usr/lib/libgnat-3.15p.so.1
#3  0x40191e3b in __gnat_malloc () from /usr/lib/libgnat-3.15p.so.1
#4  0x08049a27 in test_250076 () at test_250076.adb:13
#5  0x0804978d in main (argc=1, argv=(system.address) 0xbffffce4, 
    envp=(system.address) 0xbffffcec) at b~test_250076.adb:168
#6  0x401cddc6 in __libc_start_main () from /lib/libc.so.6

`top' indicates that the process takes 2940 MiB of virtual memory.  I
think that virtual memory has indeed been exhausted; the limit is 3072
MiB (3 GiB) for all user-space programs, with 1 GiB reserved for the
kernel.

Now, 2940 MiB / 192_650_250 iterations averages 16 bytes per
allocation.  Each Integer is theoretically only 4 bytes.  So, while
libgnat does indeed reuse previously allocated pages, there is a
non-negligible overhead.  Part of this may be due to the necessary
bookkeeping structures in the C library (malloc), but do you think
this might explain such a huge (4x) overhead?

If I remove the exception handler, I get the SIGSEGV:

Program received signal SIGSEGV, Segmentation fault.
0x40228ee3 in malloc () from /lib/libc.so.6

Back to the original question, is this normal, or should libgnat turn
the SIGSEGV into Storage_Error?

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Ludovic Brenta]

Mark H Johnson writes:
> Also - have you checked the console / system logs for any indication
> that the OOM killer is at work?

Thanks for pointing to the system log.  I hadn't thought of that.  Yes
indeed, the kernel kills the process:

kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
kernel: VM: killing process test_250076

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Randy Brukardt]

"Wojtek Narczynski" <wojtek@power.com.pl> wrote in message
news:5ad0dd8a.0405210313.70f9339d@posting.google.com...
...
> > The program allocates memory in an infinite loop using "new" and the
> > default storage pool, and of course exhausts all available storage.
> > The issue is whether or not Storage_Error should be raised.  With
> > GNAT, no exception is raised; instead the program receives SIGKILL
> > (not even SIGSEGV) from Linux.
>
> I noticed, that if the program allocates memory in 6Kb chunks or
> larger, Storate_Error is raised. If the chunk size is 5Kb or less, the
> program only prints "Killed". In between I have not tested.
>
> Also, shouldn't cases like this one, as a rule, eventually become part
> of ACATS?

There is such an ACATS test. The original version just tried to allocate
memory until it was exhausted. In our case, this caused virtual memory
target machines to thrash badly (there was about one allocated item per
page). We ended up limiting the amount of heap memory to 8 Meg in order to
make that test pass -- which really was in no one's interest. We discussed
the issues and concluded that tests that try to exhaust memory are
meaningless on virtual memory systems.

That version was replaced by a newer one that (A) allocates larger chunks of
memory; and (B) only tries to allocate a limited amount of memory before
giving up. That preserves the test for the embedded systems that it is
meaningful for, and prevents the need to do senseless things like limit the
amount of memory that you can allocate.

                    Randy.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[user]

Ludovic Brenta wrote:
> Hello,
> 
> I have received a bug report [1] against GNAT 3.15p and reproduced the
> same behaviour with GCC 3.4.  I would like the advice of language
> lawyers on the issue.
> 
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250076
> 
 > ...
 >
> PS. The original reported says that the program receives SIGSEGV.  He
> uses Linux 2.4.22.  I use 2.4.26 and the program receives SIGKILL
> instead.  I don't know if this is a known change in the kernel's
> behaviour.
> 

Hi,

I'm the original reporter of the bug. I forgot to mention that I limited 
the virtual memory of the process using 'ulimit -v 100000' in a bash.

I'm very new to Ada, but I'm impressed of all the features. In parallel 
I'm developing a little library to make C programming easier. In this 
library I malloc'ed some "emergency memory" at the program start which 
is simply freed when the process gets out of memory. Then the error 
handler can at least raise an exception or print some warning.

Perhaps the GNAT runtime library could use a similar approach.

Roland

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Ludovic Brenta]

> Hi,
>
> I'm the original reporter of the bug. I forgot to mention that I
> limited the virtual memory of the process using 'ulimit -v 100000'
> in a bash.
>
> I'm very new to Ada, but I'm impressed of all the features. In
> parallel I'm developing a little library to make C programming
> easier. In this library I malloc'ed some "emergency memory" at the
> program start which is simply freed when the process gets out of
> memory. Then the error handler can at least raise an exception or
> print some warning.
>
> Perhaps the GNAT runtime library could use a similar approach.

I just closed your bug report after determining that the issue was not
in libgnat but in the kernel.  Basically, Linux sends SIGKILL to
programs that exhaust physical RAM; as you know, SIGKILL cannot be
caught so libgnat has no chance to raise an exception.  If the virtual
memory is exhausted before physical memory is, then libgnat does raise
Storage_Error.

I am quite confident that you can implement your "emergency memory"
scheme in Ada, but this will not do anything to prevent SIGKILL.  At
best, you can recover from a SIGSEGV.  This is true also of your C
library.

BTW, why not view the Ada compiler and library as "a library to make C
programming easier"? :)

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Ludovic Brenta]

Ludovic Brenta writes:
> I am quite confident that you can implement your "emergency memory"
> scheme in Ada, but this will not do anything to prevent SIGKILL.  At
> best, you can recover from a SIGSEGV.  This is true also of your C
> library.

I forgot to point you at one of Ada's neat features called
"User-Defined Storage Pools".  Look at the Ada Reference Manual[1]
section 13.11.  You can define your own allocation and deallocation
mechanism for any access type; your storage pool may for example
preallocate X megabytes from the operating system, allocate from that
buffer, and raise Storage_Error when it is exhausted.  As long as X <=
Physical_Ram, you then have a guarantee that your porgram receives
Storage_Error and not SIGKILL.

[1] http://www.adaic.org/standards/95lrm/html/RM-TTL.html

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Wojtek Narczynski]

Hello,

> Now, 2940 MiB / 192_650_250 iterations averages 16 bytes per
> allocation.  Each Integer is theoretically only 4 bytes.  So, while
> libgnat does indeed reuse previously allocated pages, there is a
> non-negligible overhead.  Part of this may be due to the necessary
> bookkeeping structures in the C library (malloc), but do you think
> this might explain such a huge (4x) overhead?

Here is a description of how it works:

http://gee.cs.oswego.edu/dl/html/malloc.html

Indeed 16 bytes are supposed to be allocated, according to this
document.


When I actually write the allocated space, i get "Killed" for almost
any size of chunk... And yes, I do have "May 22 01:17:00 slooby
kernel: VM: killing process memory_chunk" messages in my syslog...


I would be interested in finding out wether this is specific to Linux,
and how about kernel 2.6.


Regards,
Wojtek

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Simon Wright]

Ludovic Brenta <ludovic.brenta@insalien.org> writes:

> I forgot to point you at one of Ada's neat features called
> "User-Defined Storage Pools".  Look at the Ada Reference Manual[1]
> section 13.11.  You can define your own allocation and deallocation
> mechanism for any access type; your storage pool may for example
> preallocate X megabytes from the operating system, allocate from that
> buffer, and raise Storage_Error when it is exhausted.  As long as X <=
> Physical_Ram, you then have a guarantee that your porgram receives
> Storage_Error and not SIGKILL.

The easiest way to do this is to specify the storage size for the
access type:

   type S is access ...;
   for S'Storage_Size use ...;   --  in storage elements

ALRM 3.11(18):

   If Storage_Size is specified for an access type, then the
   Storage_Size of this pool is at least that requested, and the
   storage for the pool is reclaimed when the master containing the
   declaration of the access type is left. If the implementation
   cannot satisfy the request, Storage_Error is raised at the point of
   the attribute_definition_clause. If neither Storage_Pool nor
   Storage_Size are specified, then the meaning of Storage_Size is
   implementation defined.

-- 
Simon Wright                               100% Ada, no bugs.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Duncan Sands]

> I'm very new to Ada, but I'm impressed of all the features. In parallel
> I'm developing a little library to make C programming easier. In this
> library I malloc'ed some "emergency memory" at the program start which
> is simply freed when the process gets out of memory. Then the error
> handler can at least raise an exception or print some warning.

This will not work on some operating systems (such as linux).  They don't
actually allocate the memory until you access it (in particular this means
that malloc never fails).  So you would (at least) need to fill the memory
with something after allocating it, to be sure you were really assigned some
memory.  But even so, the memory is liable to be swapped out to disk after
a while (because the OS will notice that you are not using it).  So freeing it
won't in fact give you any more memory, because that memory area is probably
already being used for something else.  And if the OOM killer (linux specific)
decides to kill you, then your program is instantly dead and you will have no
chance of freeing memory, or doing anything else for that matter.

Ciao,

Duncan.

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Martin Krischik]

Duncan Sands wrote:

>> I'm very new to Ada, but I'm impressed of all the features. In parallel
>> I'm developing a little library to make C programming easier. In this
>> library I malloc'ed some "emergency memory" at the program start which
>> is simply freed when the process gets out of memory. Then the error
>> handler can at least raise an exception or print some warning.
> 
> This will not work on some operating systems (such as linux).  They don't
> actually allocate the memory until you access it (in particular this means
> that malloc never fails). 

But malloc () will write its bookkeeping before each allocation. So if you
allocate less then a pagesize every page is used at least once.

With Regards

Martin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory conditions?

[Martin Krischik]

Robert I. Eachus wrote:

> Martin Krischik wrote:
> 
>> Just a thought: The page size on Intel CPU's is 4 kb. So most OS's on
>> Intel CPU will only handle mguarantyequests of at least one page.
>> 
>> I find it interesting that with request significantly larger then one
>> page everything seems Ok.
>> 
>> Lets Think: With GNAT allocation from the standart storrage pool are just
>> passed down to C's malloc ().
>> 
>> When smaller memory is requested most C compiler will bundele several
>> requests into one page request.
>> 
>> When larger  memory is requested some C compiler will just pass the
>> request on to the OS and some other will allways bundle.
> 
> This is interesting.  It sounds like almost the opposite of what I
> expected.  Apparently if the memory requests are large enough that some
> pages are left virtual, you get the Storage_Error.  But if the code
> results in every page allocated being 'touched' you get killed.

Of corse when you read

http://gee.cs.oswego.edu/dl/html/malloc.html

you will see that the OS service of dynamic memory on Unix is rather
primitive - compared to Windows OS/2 and such like.

The principle as I described it still holds however on Unix the program
append new memory to the end and will almost never be able to return memory
to the OS.

Unlike Windows OS/2 where a you can free an unused pages with (Dos)FreeMem.
Of corse I don't know if a particular free () implementation actually
searches for unused pages.

With Regards

Martin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory

[Robert I. Eachus]

Ludovic Brenta wrote:

> I just closed your bug report after determining that the issue was not
> in libgnat but in the kernel.  Basically, Linux sends SIGKILL to
> programs that exhaust physical RAM; as you know, SIGKILL cannot be
> caught so libgnat has no chance to raise an exception.  If the virtual
> memory is exhausted before physical memory is, then libgnat does raise
> Storage_Error.

Did you refile the bug against the kernel?  As I understand it now, the 
problem is not that the kernel raises SIGKILL if physical RAM is 
exhausted, but that malloc doesn't check that the allocation can succeed 
before making it.

Having said that, I suspect that this should be a low priority bug, and 
it will be pretty hard to fix.  It may take adding a version of malloc 
that does the check, and leaving the decision as to which one to call to 
the compiler run-time.  (GNAT should then use the new call when 
allocating storage pools, and probably continue using the current 
version for allocations in the default storage pool.

-- 

                                           Robert I. Eachus

"The terrorists rejoice in the killing of the innocent, and have 
promised similar violence against Americans, against all free peoples, 
and against any Muslims who reject their ideology of murder. Their 
barbarism cannot be appeased, and their hatred cannot be satisfied. 
There's only one way to deal with terror: We must confront the enemy and 
stay on the offensive until these killers are defeated." -- George W. Bush

Re: Is the Ada run-time required to detect out-of-memory

[Duncan Sands]

> Did you refile the bug against the kernel?  As I understand it now, the
> problem is not that the kernel raises SIGKILL if physical RAM is
> exhausted, but that malloc doesn't check that the allocation can succeed
> before making it.

I don't believe that malloc can check before making the allocation (on a system
that overcommits memory).  Maybe it can check after making the allocation though.

Duncan.

Re: Is the Ada run-time required to detect out-of-memory

[Ludovic Brenta]

"Robert I. Eachus" writes:
> Ludovic Brenta wrote:
>
>> I just closed your bug report after determining that the issue was not
>> in libgnat but in the kernel.  Basically, Linux sends SIGKILL to
>> programs that exhaust physical RAM; as you know, SIGKILL cannot be
>> caught so libgnat has no chance to raise an exception.  If the virtual
>> memory is exhausted before physical memory is, then libgnat does raise
>> Storage_Error.
>
> Did you refile the bug against the kernel?

No, I didn't.  As explained above, I leave this to the OP to decide
whether or not the current behaviour is a problem.

> As I understand it now, the problem is not that the kernel raises
> SIGKILL if physical RAM is exhausted, but that malloc doesn't check
> that the allocation can succeed before making it.
> 
> Having said that, I suspect that this should be a low priority bug,
> and it will be pretty hard to fix.  It may take adding a version of
> malloc that does the check, and leaving the decision as to which one
> to call to the compiler run-time.  (GNAT should then use the new call
> when allocating storage pools, and probably continue using the current
> version for allocations in the default storage pool.

As Duncan pointed out, a version of malloc that does the check would
be very difficult to write indeed.  It would have to hook directly
into the kernel's memory manager in order to know which pages are
swapped out, which ones are dirty, and so on.  Note that it would have
to switch paradigms in the process, from arbitrary-sized chunks to
pages and from user space to kernel space.  This also would break the
whole purpose of some important design decisions made in Linux.

Personally, I would just leave it at that.  If someone needs more
deterministic behaviour than Linux provides, they should be using a
real-time kernel without any virtual memory.

-- 
Ludovic Brenta.

Re: Is the Ada run-time required to detect out-of-memory

[Duncan Sands]

> Personally, I would just leave it at that.  If someone needs more
> deterministic behaviour than Linux provides, they should be using a
> real-time kernel without any virtual memory.

On a 2.6 kernel,

	echo 2 > /proc/sys/vm/overcommit_memory

may help.

Ciao,

Duncan.

Re: Is the Ada run-time required to detect out-of-memory

[Martin Krischik]

Robert I. Eachus wrote:

> Ludovic Brenta wrote:
> 
>> I just closed your bug report after determining that the issue was not
>> in libgnat but in the kernel.  Basically, Linux sends SIGKILL to
>> programs that exhaust physical RAM; as you know, SIGKILL cannot be
>> caught so libgnat has no chance to raise an exception.  If the virtual
>> memory is exhausted before physical memory is, then libgnat does raise
>> Storage_Error.
> 
> Did you refile the bug against the kernel?  As I understand it now, the
> problem is not that the kernel raises SIGKILL if physical RAM is
> exhausted, but that malloc doesn't check that the allocation can succeed
> before making it.

one could use calloc. calloc zeros memory and therefore touches every page.

BTW It's not a bug its a feature:

/work  Linux  martin@linux2  Mi Mai 26 08:09:24  standart
>man malloc
[....]
       Linux follows an optimistic memory allocation strategy.  This means
that when malloc() returns non-NULL  there
       is  no  guarantee  that the memory really is available. In case it
turns out that the system is out of memory,
       one or more processes will be killed by the infamous OOM killer.



GNU                         1993-04-04                  MALLOC(3)
 Manual page malloc(3) line 37/72 (END)

With Regards

Martin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory

[Duncan Sands]

> one could use calloc. calloc zeros memory and therefore touches every page.

Maybe not - perhaps the virtual memory mapping just points those pages at
the zero page (a special physical page filled with zeros), with copy-on-write
so you get assigned a real page only if you write to it (calloc won't need to
write, because it knows the page starts "filled" with zeros).

Ciao,

Duncan.

Re: Is the Ada run-time required to detect out-of-memory

[Martin Krischik]

Duncan Sands wrote:

>> one could use calloc. calloc zeros memory and therefore touches every
>> page.
> 
> Maybe not - perhaps the virtual memory mapping just points those pages at
> the zero page (a special physical page filled with zeros), with
> copy-on-write so you get assigned a real page only if you write to it
> (calloc won't need to write, because it knows the page starts "filled"
> with zeros).

Does calloc know when it gets a new page and when is gets recycled page?

After all the simplest calloc implementation would be:

void*
calloc(size_t nmemb, size_t size)
{
  auto void* result = malloc (nmemb * size);
  memset (result, '\0', nmemb * size);
  return result;
}

With Regards

Martrin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com

Re: Is the Ada run-time required to detect out-of-memory

[Duncan Sands]

> Does calloc know when it gets a new page and when is gets recycled page?

This is not really a recycled page.  In any case, the answer is probably "no".

> After all the simplest calloc implementation would be:
>
> void*
> calloc(size_t nmemb, size_t size)
> {
>   auto void* result = malloc (nmemb * size);
>   memset (result, '\0', nmemb * size);
>   return result;
> }

It's more efficient to not have to zero a page unless you write to it - in fact
to not really have any physical pages at all until you write to them.

Duncan.

Re: Is the Ada run-time required to detect out-of-memory

[Martin Krischik]

Duncan Sands wrote:

>> Does calloc know when it gets a new page and when is gets recycled page?
> 
> This is not really a recycled page.  In any case, the answer is probably
> "no".

I was thinking about pages recycled by free (). 

>> After all the simplest calloc implementation would be:
>>
>> void*
>> calloc(size_t nmemb, size_t size)
>> {
>>   auto void* result = malloc (nmemb * size);
>>   memset (result, '\0', nmemb * size);
>>   return result;
>> }
> 
> It's more efficient to not have to zero a page unless you write to it - in
> fact to not really have any physical pages at all until you write to them.

Remember: Not all memory for malloc and calloc come staid from the OS. Some
have been freed by free (); 

With Regards

Martin
-- 
mailto://krischik@users.sourceforge.net
http://www.ada.krischik.com