GNAT stuck, any idea on how to diagnose it?

GNAT stuck, any idea on how to diagnose it?

[Natasha Kerensikova]

Hello,

I have a library project, called Natools, where I accumulate
miscellaneous small stuff. I also have a test suite, and a binary called
`test_all` that runs all the test suites.

Since I'm accumulating stuff, and test_all is accessing everything, it's
getting larger and larger, and takes more and more time to build. So far
so good.

However a few days ago it seems I've reached a threshold where gnatmake
takes a much longer time to finish. I've tried letting it run, and so
far it has taken 41 CPU hours without any sign of progress, while
without my latest package test it doesn't take more than a few wallclock
minutes. Moreover, the memory usage reported by the operating system
hasn't changed between a few minutes after start and now.

Here is the command line, word-wrapped to make it easier to read:

/usr/local/gcc-aux/libexec/gcc/x86_64-aux-freebsd9.2/4.9.0/gnat1 -I
--quiet -dumpbase natools-s_expressions-templates-integers.ads -auxbase
natools-s_expressions-templates-integers -O3 -fstack-check=specific
-gnatez -gnatea -gnatec=/home/nat/code/natools/obj/GNAT-TEMP-000004.TMP
-gnatafnovy -gnateE -gnatwae -gnatA
-gnatem=/home/nat/code/natools/obj/GNAT-TEMP-000005.TMP -gnatez
-mtune=generic -march=x86-64
/home/nat/code/natools/src/natools-s_expressions-templates-integers.ads
-o /tmp//ccd4TDPm.s

Would you know what is this `gnat1`?
What part of the build process is it?
What is the command above supposed to perform?
Is it known to occasionally take a lot of time?
Can I consider it eternally stuck, or am I just too impatient?
What kind of tool can I used to help answer those questions?



Thanks in advance for your help,
Natasha

Re: GNAT stuck, any idea on how to diagnose it?

[Georg Bauhaus]

On 14.09.14 11:21, Natasha Kerensikova wrote:

> /usr/local/gcc-aux/libexec/gcc/x86_64-aux-freebsd9.2/4.9.0/gnat1 -I
> --quiet -dumpbase natools-s_expressions-templates-integers.ads -auxbase
> natools-s_expressions-templates-integers -O3 -fstack-check=specific
> -gnatez -gnatea -gnatec=/home/nat/code/natools/obj/GNAT-TEMP-000004.TMP
> -gnatafnovy -gnateE -gnatwae -gnatA
> -gnatem=/home/nat/code/natools/obj/GNAT-TEMP-000005.TMP -gnatez
> -mtune=generic -march=x86-64
> /home/nat/code/natools/src/natools-s_expressions-templates-integers.ads
> -o /tmp//ccd4TDPm.s
>
> Would you know what is this `gnat1`?

The GNAT User's Guide says this in "3.1 Compiling Programs":

    `gcc' is actually a driver program that looks at the extensions of
the file arguments and loads the appropriate compiler. For example, the
GNU C compiler is `cc1', and the Ada compiler is `gnat1'.

The User's Guide is included with the compiler, in any formats the user
may likely prefer.

> What kind of tool can I used to help answer those questions?

My first attempt would be to obtain access to one of the platforms that
have receive more extensive support for the Ada part of GCC, for example
one of the platforms supported by GPL GNAT, or Debian/GNU. This should
help with excluding issues that may need to be attributed to GNAT on FreeBSD.

Then, also, some OS tracing facility may hint at loops.
Another option is to run the translation process from a debugger.

Re: GNAT stuck, any idea on how to diagnose it?

[Dmitry A. Kazakov]

On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:

> Would you know what is this `gnat1`?
> What part of the build process is it?
> What is the command above supposed to perform?
> Is it known to occasionally take a lot of time?

Yes. It may take relatively much time on its own, but from my experience, I
can tell that GNAT consumes extremely and unreasonably much memory.

At some point it stops compiling under 32-bit systems because there is not
enough virtual memory.

You could check if your problem is caused by exhaustion of the physical
memory and turning to the swap. If swap gets involved it will never end.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: GNAT stuck, any idea on how to diagnose it?

[Brad Moore]

On 2014-09-14 8:49 AM, Dmitry A. Kazakov wrote:
> On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:
>
>> Would you know what is this `gnat1`?
>> What part of the build process is it?
>> What is the command above supposed to perform?
>> Is it known to occasionally take a lot of time?
>
> Yes. It may take relatively much time on its own, but from my experience, I
> can tell that GNAT consumes extremely and unreasonably much memory.
>
> At some point it stops compiling under 32-bit systems because there is not
> enough virtual memory.
>
> You could check if your problem is caused by exhaustion of the physical
> memory and turning to the swap. If swap gets involved it will never end.
>

I have on very rare occasion run into a case where the compiler appears 
to be stuck in an endless loop when encountering some illegal code that 
it couldn't handle. (More typically, this would involve a compiler crash 
where at least you get some diagnostic message offering a clue to the 
problem). In my case, I think it had something to do with a name 
resolution problem I think. I found the source of the problem by 
commenting out sections of new code until the
problem went away, then adding back smaller sections until the program 
reappears, until I zeroed in on the problematic code. Once I realized my
mistake, I changed the code to use legal constructs, and the code 
compiled right away.

Brad

Re: GNAT stuck, any idea on how to diagnose it?

[Robert A Duff]

Natasha Kerensikova <lithiumcat@instinctive.eu> writes:

> Since I'm accumulating stuff, and test_all is accessing everything, it's
> getting larger and larger, and takes more and more time to build. So far
> so good.

Try using the -gnatdO switch.  That's a capital Oh at the end, not zero.
It's documented in debug.adb.

- Bob

Re: GNAT stuck, any idea on how to diagnose it?

[anon]

GNAT is a command shell for Adacore's Ada compiler. And once it 
determines what action to be done, GNAT runs a number of programs.

At first GNAT may call the GNAT1 (Ada compiler). This program converts 
Ada to assembly language, if the source code is correct, else reports
errors.

   "gnat compile <compiler options> <source filename>.ad[b|s]"
or
   "gnat make <compiler options> <source filename>.ad[b|s]"

preforms
   GNAT --> "gcc <compiler options> <source filename>.ad[b|s]"
    gcc --> "gnat1 <compiler options> <source filename>.ad[b|s]"

Second, if GNAT1 returns a no errors condition then GNAT calls the gcc
assembler "as" (gcc assembler)

   GNAT --> "as <assembly options> <source filename>"

You can bypass GNAT and the "gcc" steps, by using

    gnat1 <compiler options> <source filename>.ad[b|s]
    as <assembly options> <source filename>.s

Also, GNAT BIND works almost the same way as GNAT COMPILE except 
BIND creates either a Ada package (default) or a C binding package
that can be linked to the runtime code. Note: The binder creates 
a package which starts with "b~" added to the source filename.

   GNAT --> "gnatbind <binder options> <source filename>.ali"
   GNAT --> "gcc <binder options> <b~source filename>.adb"
    gcc --> "gnat1 <binder options> <b~source filename>.adb"
   GNAT --> "as <assembly options> <b~source filename>.s"

bypassing GNAT would be:

    gnatbind <bind options> <source filename>.ali
    gnat1 <bind options> options> <b~source filename>.adb
    as <assembly options> <b~source filename>.s



But as for time GNAT does not take that much time.  The part that takes 
the longest is the gcc backend which GNAT uses to convert it internal 
gcc IR language to assembly.  Mostly it is the optimization phase
in the gcc backend that requires more time ( options -O2 (default), 
-O3, -O4 ).

now the process that GNAT1 uses is 

        GNAT1       :==  Ada source  -->  AST  -->  GCC IR 

at this movement the GCC backend takes over and processes the GCC IR 
into assembly

        gcc backend :==  GCC IR      -->  ???  -->  assembly code


For speed, GNAT1 loads the entire code into memory at one time. Then 
basically processes the source as a large data array, while others 
compilers may read a line of code at a time (reduced speed).  With all 
of the checks that the Ada requires in compiling, GNAT1 is very 
efficiency during the syntactic and semantic phases. Now the slow
down starts to occur in the backend converter "gigi" which is written 
in C. The "gigi" uses the GCC backend which is efficiency for C/C++ but 
is not as efficiency for other languages like Ada, Fortran, etc. 

One explanation is in translating Ada into the "GCC IR" code the 
"gigi" may use or alter the code that makes it difficult for the 
GCC optimization to process.

Which means that Adacore's GNAT by now (20 years) should have its own 
backend written in Ada, specifically for Ada. That way the world could 
see the true speed of the GNAT compiler.


In <vlxfpcmsetjg$.1em03ijtae3i6$.dlg@40tude.net>, "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:
>
> Would you know what is this `gnat1`?
> What part of the build process is it?
> What is the command above supposed to perform?
> Is it known to occasionally take a lot of time?

Re: GNAT stuck, any idea on how to diagnose it?

[Natasha Kerensikova]

Hello,

On 2014-09-14, Brad Moore <brad.moore@shaw.ca> wrote:
> On 2014-09-14 8:49 AM, Dmitry A. Kazakov wrote:
>> On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:
> I have on very rare occasion run into a case where the compiler appears 
> to be stuck in an endless loop when encountering some illegal code that 
> it couldn't handle. (More typically, this would involve a compiler crash 
> where at least you get some diagnostic message offering a clue to the 
> problem). In my case, I think it had something to do with a name 
> resolution problem I think. I found the source of the problem by 
> commenting out sections of new code until the
> problem went away, then adding back smaller sections until the program 
> reappears, until I zeroed in on the problematic code. Once I realized my
> mistake, I changed the code to use legal constructs, and the code 
> compiled right away.

It turns out your message was the largest contributor to my diagnostic,
except I don't see any mistake in my code (though thinking about it now,
I seem to remember people advising against expression functions
returning unconstrained values).

The presumably-endless loop was triggered by the following expression:

   (if Ref.Is_Empty then To_Atom (Fallback) else Ref.Query.Data.all);

It was originally in an expression function, but turning it into a
normal function whose body is only a return statement with the
expression above leads to the same symptoms.

However, turning it into an if statement in the normal function below
sloves the problem:

   if Ref.Is_Empty then
      return To_Atom (Fallback);
   else
      return Ref.Query.Data.all;
   end if;

In case anyone is interested, the full code is published there
https://github.com/faelys/natools/blob/trunk/src/natools-s_expressions-templates-generic_integers.adb#L291

Now a secondary question would be, how long is it reasonable to let GNAT
run before assuming it won't terminate ever?
My 41h here was probably extremely conservative...


Thank you all for your help,
Natasha

Re: GNAT stuck, any idea on how to diagnose it?

[Anh Vo]

On Sunday, September 14, 2014 2:47:47 PM UTC-7, Natasha Kerensikova wrote:
> Hello,
>  
> On 2014-09-14, Brad Moore <brad.moore@shaw.ca> wrote: 
> > On 2014-09-14 8:49 AM, Dmitry A. Kazakov wrote: 
> >> On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:  
> 
> Now a secondary question would be, how long is it reasonable to let GNAT 
> run before assuming it won't terminate ever?

It depends on size of the code in question. For normal case if gnat1 occupies close to 99% CPU in 30 seconds or more, that means the compiler runs away. I would submit a bug report.

A. Vo

Re: GNAT stuck, any idea on how to diagnose it?

[Dmitry A. Kazakov]

On Sun, 14 Sep 2014 15:56:35 -0700 (PDT), Anh Vo wrote:

> On Sunday, September 14, 2014 2:47:47 PM UTC-7, Natasha Kerensikova wrote:
>> Hello,
>>  
>> On 2014-09-14, Brad Moore <brad.moore@shaw.ca> wrote: 
>>> On 2014-09-14 8:49 AM, Dmitry A. Kazakov wrote: 
>>>> On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:  
>> 
>> Now a secondary question would be, how long is it reasonable to let GNAT 
>> run before assuming it won't terminate ever?
> 
> It depends on size of the code in question. For normal case if gnat1
> occupies close to 99% CPU in 30 seconds or more, that means the compiler
> runs away. I would submit a bug report.

I would say 1 hour for the compiler and 1/2 for the linker (which could be
very slow too). 

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: GNAT stuck, any idea on how to diagnose it?

[Natasha Kerensikova]

Hello,

On 2014-09-14, Dmitry A. Kazakov <mailbox@dmitry-kazakov.de> wrote:
> On Sun, 14 Sep 2014 09:21:18 +0000 (UTC), Natasha Kerensikova wrote:
>
>> Would you know what is this `gnat1`?
>> What part of the build process is it?
>> What is the command above supposed to perform?
>> Is it known to occasionally take a lot of time?
>
> Yes. It may take relatively much time on its own, but from my experience, I
> can tell that GNAT consumes extremely and unreasonably much memory.
>
> At some point it stops compiling under 32-bit systems because there is not
> enough virtual memory.
>
> You could check if your problem is caused by exhaustion of the physical
> memory and turning to the swap. If swap gets involved it will never end.

As I wrote in the OP, the memory used by gnat1 process, as reported by
the operating system, was constant. Actual numbers depend on what is
measured, as is often the case in modern OS, but all of them where
constant, from 72 MB of physical memory to about 140 MB of addressable
space.

From there, can we deduce that 100% CPU usage with constant memory usage
means the compiler is stuck?

Or are there parts of the compiling process that can legitimately a
significant amount of time without using extra memory?



Thanks for you comments,
Natasha

Re: GNAT stuck, any idea on how to diagnose it?

[Dmitry A. Kazakov]

On Sun, 14 Sep 2014 21:45:16 +0000 (UTC), anon@att.net wrote:

> One explanation is in translating Ada into the "GCC IR" code the 
> "gigi" may use or alter the code that makes it difficult for the 
> GCC optimization to process.

Interesting.

My impression based on how I dealt with the problem was that it was not the
compiler or the code generator which grabbed memory. Maybe it was the
symbols cross reference and other stuff that helped generating error
messages and prompts. For example, making a separate body or splitting off
a child package does not help. I seems that it must be a distance of two or
more packages apart from the offending body. So my wild guess was that GNAT
grabbed all symbols from all packages at the distance 1, even from the
invisible ones and their bodies or ones referenced from the bodies to make
suggestions on an error, but did not do that for the packages at the
distance 2.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: GNAT stuck, any idea on how to diagnose it?

[Jeffrey Carter]

On 09/15/2014 12:16 AM, Natasha Kerensikova wrote:
> 
> From there, can we deduce that 100% CPU usage with constant memory usage
> means the compiler is stuck?

I think it's clear that you've found an error in the translation of if expressions.

-- 
Jeff Carter
"Go and boil your bottoms."
Monty Python & the Holy Grail
01

Re: GNAT stuck, any idea on how to diagnose it?

[anon]

To stop the call to gigi and the GCC backend. One can use the debug 
option "-gnatdH"  the "dH" must be lower case 'd' and uppercase 'H'. 
Even though this option can be used on "gnat1" or "gnat compile" or 
even "gnat make". It should be limited to "gnat1" or "gnat compile".

The compiler will generate a <source file>.ali and <source file>.s.
The <source file>.s will only contain a 3 line header for a gcc assembly 
file.

Now all of GNAT debug options can be found in compiler source package 
"debug" and a very simple explanation of each option in the file 
"debug.adb". 

Also, for those options that generate text output. Some of those 
can generate 100s of lines, even for a simple hello world program.
An example is -gnatdt" outputs the a detailed AST listing.

Note: Some of the text debug option will also conflict with other 
      text debug options. So, you need to start slowly (one option 
      at a time) until you learn which options you need and which 
      option conflict with others.


In <mmnjpohtp615.1d24bw7p0fns8.dlg@40tude.net>, "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>On Sun, 14 Sep 2014 21:45:16 +0000 (UTC), anon@att.net wrote:
>
>> One explanation is in translating Ada into the "GCC IR" code the 
>> "gigi" may use or alter the code that makes it difficult for the 
>> GCC optimization to process.
>
>Interesting.
>
>My impression based on how I dealt with the problem was that it was not the
>compiler or the code generator which grabbed memory. Maybe it was the
>symbols cross reference and other stuff that helped generating error
>messages and prompts. For example, making a separate body or splitting off
>a child package does not help. I seems that it must be a distance of two or
>more packages apart from the offending body. So my wild guess was that GNAT
>grabbed all symbols from all packages at the distance 1, even from the
>invisible ones and their bodies or ones referenced from the bodies to make
>suggestions on an error, but did not do that for the packages at the
>distance 2.
>
>-- 
>Regards,
>Dmitry A. Kazakov
>http://www.dmitry-kazakov.de