Hint to Ada newbies

Hint to Ada newbies

[reinkor]

Just sharing experience:

I tried gnat compiler option "-gnatwa" and revealed a lot of weakness in my Ada project. It also helped to improve the structure of the code which I have worked on and modified over time.

Just a hint to newbies :-)

https://gcc.gnu.org/onlinedocs/gnat_ugn/Warning-Message-Control.html

reinert

Re: Hint to Ada newbies

[Per Sandberg]

Som more hints on switches:

This is a snippet of my usual project file during development. It will 
force me to keep a consistent layout and catch a lot of errors during 
compile and tests.

my.gpr:
------------------------------------
project My is

    for Source_Dirs use ("src");
    for Object_Dir use ".obj";
    for Main use ("main.adb");

   package Builder is
     for Default_Switches("Ada") use ("-k", "-j0");
     for Global_Configuration_Pragmas use project'Project_Dir & "my.gpp";
     --  Initialize data to bad values.
   end Builder;

   package Compiler is
       for Switches ("ada") use
         ("-gnatyybcfhiklnprtu", "-gnatyN256", "-gnaty3", -- Check 
layout.
          "-gnatVa", --  Turn on all validity checking options
          "-gnatwa", --  Turn most info/Warning
          "-gnatwe", --  Treat all warnings as errors
          "-gnata",  --  Enable Assertions.
          "-gnateE", --  Generate extra information in exception 
messages
          "-gnatQ"); --  Don't quit, write ali/tree file even if compile 
errors
    end Compiler;

    package Binder is
       for Switches ("ada") use
         ("-E");  -- Store callstack in exceptions.
    end Binder;
end Test_My;
------------------------------------
my.gpp
----------------------------------
pragma Initialize_Scalars;
--------------------------------

/P


Den 2017-02-09 kl. 05:17, skrev reinkor:
> Just sharing experience:
>
> I tried gnat compiler option "-gnatwa" and revealed a lot of weakness in my Ada project. It also helped to improve the structure of the code which I have worked on and modified over time.
>
> Just a hint to newbies :-)
>
> https://gcc.gnu.org/onlinedocs/gnat_ugn/Warning-Message-Control.html
>
> reinert
>

Re: Hint to Ada newbies

[Simon Wright]

Per Sandberg <per.s.sandberg@bahnhof.se> writes:

>   package Builder is
>     for Default_Switches("Ada") use ("-k", "-j0");

I have had to use the power switch to stop a build using -j0! '-j<1 less
than the number of cores>' does pretty well.

>   package Compiler is
>       for Switches ("ada") use
>         ("-gnatyybcfhiklnprtu", "-gnatyN256", "-gnaty3", -- Check
> layout.

(a) personally I use -gnaty on its own (equivalent of
    -gnaty3abcefhiklmnprst, run 'gnatmake -h' for the switches). If I
    really didn't want e.g. the standard -gnatye (check end/exit labels
    present), I think I'd say "-gnaty -gnaty-e".

(b) -gnatyN256? M256 I could understand, if you don't mind unreadably
    long lines, N means "turn off all checks", N256 is undefined.

Re: Hint to Ada newbies

[Robert A Duff]

Simon Wright <simon@pushface.org> writes:

> I have had to use the power switch to stop a build using -j0! '-j<1 less
> than the number of cores>' does pretty well.

Which operating system?

- Bob

Re: Hint to Ada newbies

[Dmitry A. Kazakov]

On 10/02/2017 00:12, Robert A Duff wrote:
> Simon Wright <simon@pushface.org> writes:
>
>> I have had to use the power switch to stop a build using -j0! '-j<1 less
>> than the number of cores>' does pretty well.
>
> Which operating system?

Windows in my case, when compiling native, cross or under a virtual 
machine. But there is no need to reset. One should just wait a few 
minutes and then it would go again.

I always change the priority of gprbuild to "below normal" in the task 
manager (and one of the virtual machine too) in order to prevent locks 
like that.

It is not AdaCore to blame, IMO.

A more grave problem is when there are free cores and no free memory. 
This leads to swapping which may require hours if not days to wait 
before the OS becomes responsible again. I had this on ARM Linux quite 
frequently. Maybe this is what Simon had too.

P.S. It would be nice to have a smarter kind of -j0 switch:

1. No new instances if memory or CPU use is greater than some limit 
(e.g. 80%).

2. Spawned instances run on the idle priority.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Hint to Ada newbies

[Hadrien Grasland]

Le vendredi 10 février 2017 10:22:56 UTC+1, Dmitry A. Kazakov a écrit :
> On 10/02/2017 00:12, Robert A Duff wrote:
> > Simon Wright writes:
> >
> >> I have had to use the power switch to stop a build using -j0! '-j<1 less
> >> than the number of cores>' does pretty well.
> >
> > Which operating system?
> 
> Windows in my case, when compiling native, cross or under a virtual 
> machine. But there is no need to reset. One should just wait a few 
> minutes and then it would go again.
> 
> I always change the priority of gprbuild to "below normal" in the task 
> manager (and one of the virtual machine too) in order to prevent locks 
> like that.
> 
> It is not AdaCore to blame, IMO.
> 
> A more grave problem is when there are free cores and no free memory. 
> This leads to swapping which may require hours if not days to wait 
> before the OS becomes responsible again. I had this on ARM Linux quite 
> frequently. Maybe this is what Simon had too.
> 
> P.S. It would be nice to have a smarter kind of -j0 switch:
> 
> 1. No new instances if memory or CPU use is greater than some limit 
> (e.g. 80%).
> 
> 2. Spawned instances run on the idle priority.

The difficult part with #1 is that make cannot easily predict in advance how much memory a newly spawned build process will take, and which relative share of the host system's RAM that will be. So if such a heuristic existed, it should be tunable to some user-specified percentage.

In addition, in C++ at least (don't know in Ada), linking is a lot more memory-intensive than compilation in large projects. So the entity spawning processes should probably have different thresholds for compilation and linking tasks.

In the end, the following simple strategy works relatively well for me:

- Disable swap and rely on the Linux OOM killer to dispose of runaway build processes. With the large amount of fast RAM that we have in modern computers, swap is useless in the common case where memory is plentiful, and brings the system down to a near-halted state in the uncommon case where the system runs out of memory.
- Try to build with maximal concurrency. On a well-dimensioned build machine with at least 2GB of RAM per CPU core, this works well most of the time.
- If the build is killed by the Linux OOM killer, restart it with less processes.

Re: Hint to Ada newbies

[Dmitry A. Kazakov]

On 2017-02-11 10:16, Hadrien Grasland wrote:
> Le vendredi 10 février 2017 10:22:56 UTC+1, Dmitry A. Kazakov a écrit :
>> On 10/02/2017 00:12, Robert A Duff wrote:
>>> Simon Wright writes:
>>>
>>>> I have had to use the power switch to stop a build using -j0! '-j<1 less
>>>> than the number of cores>' does pretty well.
>>>
>>> Which operating system?
>>
>> Windows in my case, when compiling native, cross or under a virtual
>> machine. But there is no need to reset. One should just wait a few
>> minutes and then it would go again.
>>
>> I always change the priority of gprbuild to "below normal" in the task
>> manager (and one of the virtual machine too) in order to prevent locks
>> like that.
>>
>> It is not AdaCore to blame, IMO.
>>
>> A more grave problem is when there are free cores and no free memory.
>> This leads to swapping which may require hours if not days to wait
>> before the OS becomes responsible again. I had this on ARM Linux quite
>> frequently. Maybe this is what Simon had too.
>>
>> P.S. It would be nice to have a smarter kind of -j0 switch:
>>
>> 1. No new instances if memory or CPU use is greater than some limit
>> (e.g. 80%).
>>
>> 2. Spawned instances run on the idle priority.
>
> The difficult part with #1 is that make cannot easily predict in
> advance how much memory a newly spawned build process will take, and
> which relative share of the host system's RAM that will be. So if such a
> heuristic existed, it should be tunable to some user-specified percentage.

One problem is the OS design flaw that requested memory is not actually 
claimed before being actually accessed. It leads to allocation of 
non-existent memory. The major rule of SW design violated: faults must 
be reported as early as possible.

> In addition, in C++ at least (don't know in Ada), linking is a lot
> more memory-intensive than compilation in large projects. So the entity
> spawning processes should probably have different thresholds for
> compilation and linking tasks.

It is not only C++. GNU linker performance is catastrophic for Ada too.

> In the end, the following simple strategy works relatively well for me:
>
> - Disable swap and rely on the Linux OOM killer to dispose of
> runaway  build processes. With the large amount of fast RAM that we have in
> modern computers, swap is useless in the common case where memory is
> plentiful, and brings the system down to a near-halted state in the
> uncommon case where the system runs out of memory.
> - Try to build with maximal concurrency. On a well-dimensioned build
> machine with at least 2GB of RAM per CPU core, this works well most of
> the time.
> - If the build is killed by the Linux OOM killer, restart it with less processes.

It would not help if there is not enough memory to compile. The way GNAT 
works it may require 3GB or more to compile a project. On multiple 
occasions I had to restructure my project in order to reduce memory 
consumption. It is tricky because that does not depend on the 
compilation unit size.

I think a more effective strategy would be to freeze all GCC instances 
but one once swapping occurs. Upon the instance completion another one 
is resumed. When the last one ends one can return to the default procedure.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Hint to Ada newbies

[Simon Wright]

Robert A Duff <bobduff@TheWorld.com> writes:

> Simon Wright <simon@pushface.org> writes:
>
>> I have had to use the power switch to stop a build using -j0! '-j<1 less
>> than the number of cores>' does pretty well.
>
> Which operating system?

macOS (actually a few releases ago, Mavericks maybe). And it's just
possible it was the equivalent switch in Make.

Re: Hint to Ada newbies

[Hadrien Grasland]

Le vendredi 10 février 2017 10:56:53 UTC+1, Simon Wright a écrit :
> Robert A Duff writes:
> 
> > Simon Wright writes:
> >
> >> I have had to use the power switch to stop a build using -j0! '-j<1 less
> >> than the number of cores>' does pretty well.
> >
> > Which operating system?
> 
> macOS (actually a few releases ago, Mavericks maybe). And it's just
> possible it was the equivalent switch in Make.

There's actually a major difference between gnatmake's -j0 and make's -j switches.

gnatmake -j0 means "Try to keep all my CPU cores busy", i.e. use 4 processes on a 4-core machine. This is reasonable, as long as you have enough RAM to handle it.

make -j means "Spawn new processes as soon as you can", i.e. use an unbounded amount of processes. Not only is this strategy stupid (your build will actually run slower if too many processes compete for system resources), it is also the easiest way to get your machine to run out of memory.

make -l ("Do not spawn new processes if the CPU load is higher than x") was an attempt to address this broken design, which fails because it cannot handle the system load inertia: stating a new process may only increase the CPU load a couple of seconds later, and this is enough for make to spawn tons of build processes in meantime.

Re: Hint to Ada newbies

[reinkor]

>    package Compiler is
>        for Switches ("ada") use
>          ("-gnatyybcfhiklnprtu", "-gnatyN256", "-gnaty3", -- Check layout.

I often use Pretty_Printer defaults like this:

   package Pretty_Printer is
      for Default_Switches ("ada") use ("-c2", "-l2", "-c3", "-c4");
   end Pretty_Printer;

This seems to be conflicting with your layout check. 
Any proposal for Pretty_Printer Default_Switches ?

reinert

Re: Hint to Ada newbies

[John Smith]

Thanks, this is super useful.

On Thursday, February 9, 2017 at 1:06:32 AM UTC-5, Per Sandberg wrote:
> Som more hints on switches:
> 
> This is a snippet of my usual project file during development. It will 
> force me to keep a consistent layout and catch a lot of errors during 
> compile and tests.
> 
> my.gpr:
> ------------------------------------
> project My is
> 
>     for Source_Dirs use ("src");
>     for Object_Dir use ".obj";
>     for Main use ("main.adb");
> 
>    package Builder is
>      for Default_Switches("Ada") use ("-k", "-j0");
>      for Global_Configuration_Pragmas use project'Project_Dir & "my.gpp";
>      --  Initialize data to bad values.
>    end Builder;
> 
>    package Compiler is
>        for Switches ("ada") use
>          ("-gnatyybcfhiklnprtu", "-gnatyN256", "-gnaty3", -- Check 
> layout.
>           "-gnatVa", --  Turn on all validity checking options
>           "-gnatwa", --  Turn most info/Warning
>           "-gnatwe", --  Treat all warnings as errors
>           "-gnata",  --  Enable Assertions.
>           "-gnateE", --  Generate extra information in exception 
> messages
>           "-gnatQ"); --  Don't quit, write ali/tree file even if compile 
> errors
>     end Compiler;
> 
>     package Binder is
>        for Switches ("ada") use
>          ("-E");  -- Store callstack in exceptions.
>     end Binder;
> end Test_My;
> ------------------------------------
> my.gpp
> ----------------------------------
> pragma Initialize_Scalars;
> --------------------------------
> 
> /P
> 
> 
> Den 2017-02-09 kl. 05:17, skrev reinkor:
> > Just sharing experience:
> >
> > I tried gnat compiler option "-gnatwa" and revealed a lot of weakness in my Ada project. It also helped to improve the structure of the code which I have worked on and modified over time.
> >
> > Just a hint to newbies :-)
> >
> > https://gcc.gnu.org/onlinedocs/gnat_ugn/Warning-Message-Control.html
> >
> > reinert
> >

Re: Hint to Ada newbies

[stevenselectronicmail]

On Wednesday, February 8, 2017 at 8:17:45 PM UTC-8, reinkor wrote:
> Just sharing experience:
> 
> I tried gnat compiler option "-gnatwa" and revealed a lot of weakness in my Ada project. It also helped to improve the structure of the code which I have worked on and modified over time.
> 
> Just a hint to newbies :-)
> 
> https://gcc.gnu.org/onlinedocs/gnat_ugn/Warning-Message-Control.html
> 
> reinert

I prefer using pragmas for these.

pragma Initialize_Scalars;
pragma Assertion_Policy (CHECK);
pragma Debug_Policy (CHECK);
pragma Check_Float_Overflow;
pragma Overflow_Mode (Strict);
pragma Unsuppress (Overflow_Check);
pragma Validity_Checks (ALL_CHECKS);

pragma Prefix_Exception_Messages;

pragma Style_Checks (On);
pragma Style_Checks ("3aAbCdefiIklnprsStu");

pragma Warnings (On);
pragma Warnings ("ah.i.ktFK");