chopping Ada source that have preprocessor symbols in them

chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

I need to integrate an Ada source program that consists
of a few compilation units. When I want one unit per file,
then, usually, I can call gnatchop and be done.

However, the program is using preprocessor symbols that
I wish to keep. gnatchop can parse source files using
different characters sets, but apparently it cannot parse
files that use preprocessor symbols, including GNAT ones.

My "solutions" are not solutions, I feel. Either

- I call gnatprep with the original program as input and use
   the output as input to gnatchop. I loose the preprocessor
    symbols in all resulting compilation units,

or

- I do manual labor.

Neither is working well, really. Are there other options?

Re: chopping Ada source that have preprocessor symbols in them

[codeallergy]

Hi Georg,

gnatchop call gcc with the -gnatu flag to generates the unit separation offsets. example

gcc -c -gnats -gnatu -x ada pack_test.ada
Unit pack_test (spec) line 1, file offset 0, file name pack_test.ads
Unit pack_test (body) line 10, file offset 90, file name pack_test.adb
End of compilation

the problem is that the offsets are for the preprocessed file but gnatchop separates the non preprocessed one.
you could use gnatchop without any preprocessor flags (you will get errors but the file should be cut rightly).

    - Romcgb

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 06.02.13 06:29, codeallergy wrote:
> Hi Georg,
>
> gnatchop call gcc with the -gnatu flag to generates the unit separation offsets.

Interesting switch.

> the problem is that the offsets are for the preprocessed file but gnatchop separates the non preprocessed one.
> you could use gnatchop without any preprocessor flags (you will get errors but the file should be cut rightly).

Some parse errors will stop gnatchop, depending on where
the preprocessor symbols are, so I cannot ignore the errors,
unfortunately. I noticed that gnatchop accepts -gnatep
(but not -P, o.K.). Using -gnatep makes the problems appear
exactly as you indicated. Gnatchop cuts the Ada source program
at the wrong offsets. Test case:

$ gnatchop -w -gnatep=multiple_units.ep multiple_units.ada Source
splitting multiple_units.ada into:
    Source/unit_1.ads
    Source/unit_2.ads
    Source/unit_3.ads
$ ls Source/unit_?.ads | while read f ; do cat $f ; echo "<<<*>>>" ; done
package Unit_1 is
    type T is tagged null record;
    procedure Op (Item : T);
end Unit_1;
<<<*>>>

with Unit_1;
$prv with Ada.Containers.Vectors;
package Unit_2 is
    type T is new Unit_1.T with null record;
    $ovr
    procedure Op (Item : T);
private
    package Foo is new Ada.Containers.Vectors (Positive, Character);
end Unit_2;

with Uni<<<*>>>
t_1;
package Unit_3 is
end Unit_3;
<<<*>>>

$ cat multiple_units.ep
* "multiple_units.prep"
$ cat multiple_units.prep
ovr := overriding
prv := private
$ cat multiple_units.ada
package Unit_1 is
    type T is tagged null record;
    procedure Op (Item : T);
end Unit_1;

with Unit_1;
$prv with Ada.Containers.Vectors;
package Unit_2 is
    type T is new Unit_1.T with null record;
    $ovr
    procedure Op (Item : T);
private
    package Foo is new Ada.Containers.Vectors (Positive, Character);
end Unit_2;

with Unit_1;
package Unit_3 is
end Unit_3;
$

Thanks for the pointers. I wonder if this is a bug or a feature?

Re: chopping Ada source that have preprocessor symbols in them

[Stephen Leake]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> - I call gnatprep with the original program as input and use
>   the output as input to gnatchop. I loose the preprocessor
>    symbols in all resulting compilation units,

What's wrong with this option? With the right makefile rules, it should
be be fully automated. Why is "losing the preprocessor symbols" bad?
They get "lost" either way.

-- 
-- Stephe

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 08.02.13 13:22, Stephen Leake wrote:
> Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:
> 
>> - I call gnatprep with the original program as input and use
>>   the output as input to gnatchop. I loose the preprocessor
>>    symbols in all resulting compilation units,
> 
> What's wrong with this option?

When I want to use GNAT's file based approach, *and* also want to
continue preprocessing with GNATprep or gcc in order to handle
different configurations/language support/..., then I need the
symbols to stay in the chopped source. After this step, I could use
-gnatep or Makefile rules as necessary.

Re: chopping Ada source that have preprocessor symbols in them

[Florian Weimer]

* Georg Bauhaus:

> However, the program is using preprocessor symbols that
> I wish to keep. gnatchop can parse source files using
> different characters sets, but apparently it cannot parse
> files that use preprocessor symbols, including GNAT ones.

Given that the choice of preprocessor symbols can change how the file
is split up, I think what you want is not solvable in general.

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 08.02.13 21:27, Florian Weimer wrote:
> * Georg Bauhaus:
> 
>> However, the program is using preprocessor symbols that
>> I wish to keep. gnatchop can parse source files using
>> different characters sets, but apparently it cannot parse
>> files that use preprocessor symbols, including GNAT ones.
> 
> Given that the choice of preprocessor symbols can change how the file
> is split up, I think what you want is not solvable in general.

True, but GNAT's preprocessor symbols should keep things
reasonably sane? Unless the programmer chooses to define
substitutes for words like "package", say, in which case
the preprocesor will have to know the substitute. Not for
actually replacing, but for deciding where gnatchop should
should chop.

For reference,

  Definition -> Symbol ":=" Value
  Symbol -> identifier_ASCII
  Value  -> ε
  Value  -> string_literal
  Value  -> 1*{ letter, digit, '.', '_' }

Re: chopping Ada source that have preprocessor symbols in them

[Stephen Leake]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> On 08.02.13 13:22, Stephen Leake wrote:
>> Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:
>> 
>>> - I call gnatprep with the original program as input and use
>>>   the output as input to gnatchop. I loose the preprocessor
>>>    symbols in all resulting compilation units,
>> 
>> What's wrong with this option?
>
> When I want to use GNAT's file based approach, *and* also want to
> continue preprocessing with GNATprep or gcc in order to handle
> different configurations/language support/..., then I need the
> symbols to stay in the chopped source. 

Why?

I must be missing something.

You have a file in CM:

a_file.ada.gp

It needs to be processed by gnatprep, then by gnatchop, and finally by
gprbuild. So you have rules in your makefile that do that:

gnatprep a_file.ada.gp => a_file.ada

gnatchop a_file.ada => a_file_1.ads, a_file_2.adb ...

gprbuild a_file_3.adb

only a_file.ada.gp is in CM: all the other files are temporaries.

If you need to start over and specify different options for gnatprep,
you just do that, and repeat all the steps.

What is the problem?

-- 
-- Stephe

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 09.02.13 03:51, Stephen Leake wrote:
> You have a file in CM:
>
> a_file.ada.gp
>
> It needs to be processed by gnatprep, then by gnatchop, and finally by
> gprbuild.

The problem is that running the chain of tools again and again is
prohibitively inefficient.  It boils down to recompiling the world
after each edit.

The full picture is this: I am given, and will give back, a file,

a_file.ada.gp

It has about two dozen compilation units in it, some of them have many
generic instantiations that will keep the compiler busy. I need to
edit just some unit and then compile the program again.

There are two "configurations", C1, and C2. Starting in C1,
suppose I first run gnatprep on a_file.ada.gp, then gnatchop the
preprocessed output. The two steps will produce about two dozen files.
I can then call gnatmake as usual.

Next I want to edit unit U1. There now is a file F1, fully preprocessed and
chopped. Again, I can run gnatmake as usual and the compiler will
recompile only the units that depend on F1.

To switch configurations from C1 to C2, I'd need to go back
to the original a_file.ada.gp, which lacks the edits. Edit,
or concatenate. Then gnatprep, gnatchop, and compile.

If chopping would preserve the preprocessor symbols in the output
files, I could switch conigurations after edits without having to
(re)construct a_file.ada.gp and starting the process from scratch.

(An Ada implementation that has a detailed mapping between compilation
units and files of source text seems a bit more flexible here.)

Directories to the rescue? Not quite, because editing unit F1 in one
directory for C1 will not adjust F1 in another directory for C2.

When I can keep the preprocessor symbols in the chopped output, then
then gcc -gnatep=C1.gp, will work as usual after edits,
i.e. it will only compile files as necessary.

A Makefile seems good for switching configurations, though, or
recreating a_file.ada.gp from chopped and edited unit files.

Re: chopping Ada source that have preprocessor symbols in them

[Simon Wright]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> The problem is that running the chain of tools again and again is
> prohibitively inefficient.  It boils down to recompiling the world
> after each edit.

You could try the gnatmake (and, I think, gprbuild) flag -m:

gnatmake:
   Specify that the minimum necessary amount of recompilations be
   performed. In this mode gnatmake ignores time stamp differences when
   the only modifications to a source file consist in adding/removing
   comments, empty lines, spaces or tabs. This means that if you have
   changed the comments in a source file or have simply reformatted it,
   using this switch will tell gnatmake not to recompile files that
   depend on it (provided other sources on which these files depend have
   undergone no semantic modifications). Note that the debugging
   information may be out of date with respect to the sources if the -m
   switch causes a compilation to be switched, so the use of this switch
   represents a trade-off between compilation time and accurate
   debugging information.

Re: chopping Ada source that have preprocessor symbols in them

[Björn Persson]

Georg Bauhaus wrote:
> I need to integrate an Ada source program that consists
> of a few compilation units. When I want one unit per file,
> then, usually, I can call gnatchop and be done.
> 
> However, the program is using preprocessor symbols that
> I wish to keep. gnatchop can parse source files using
> different characters sets, but apparently it cannot parse
> files that use preprocessor symbols, including GNAT ones.
> 
> My "solutions" are not solutions, I feel. Either
> 
> - I call gnatprep with the original program as input and use
>    the output as input to gnatchop. I loose the preprocessor
>     symbols in all resulting compilation units,
> 
> or
> 
> - I do manual labor.
> 
> Neither is working well, really. Are there other options?

I take it you need to keep the original source file up to date and chop
it repeatedly, otherwise you would have simply done the manual labor by
now. Perhaps you could do a reversible transformation of the file to
make the preprocessor symbols look like Ada identifiers that Gnatchop
will accept? Something like this:

#!/bin/sh
sed 's/\$/My_Special_Dollar_Replacement/g' a_file.ada.gp >a_file.munged
gnatchop -w a_file.munged
sed -i s/My_Special_Dollar_Replacement/$/g *.ad?

Or replace the last line with a loop if your Sed doesn't support
in-place editing.

While this hack is certainly possible to break I think it should work
well enough in practice. I think I would prefer to use source reference
pragmas and do my editing in the actual source file though, unless the
compilation time is really prohibitive even with minimal recompilation.

Björn Persson

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 09.02.13 21:15, Simon Wright wrote:
> Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:
>
>> The problem is that running the chain of tools again and again is
>> prohibitively inefficient.  It boils down to recompiling the world
>> after each edit.
>
> You could try the gnatmake (and, I think, gprbuild) flag -m:

Promising, but there is a catch.

As Björn said, the compiler should ideally be able to refer to the
original source file as usual. Now, while switch -m does have the desired
effect when chaining the three steps gnatprep && gnatchop && gnatmake,
the compiler messages will in the end refer to the intermediate file
that gnatprep will have created. Not ideal, unless I can convince
the IDE that it should automatically substitute the original file's
name for the name of the intermediate file when inspecting the
compiler's diagnostics. A kludge, but maybe a possibility.

The alternative switch -gnatep would obviate the need for gnatprep
but will, alas, cancel the effect of switch -m, AFAICS. (In cases
when gnatchop will compute usable offsets.)

Re: chopping Ada source that have preprocessor symbols in them

[Björn Persson]

Georg Bauhaus wrote:
> As Björn said, the compiler should ideally be able to refer to the
> original source file as usual. Now, while switch -m does have the desired
> effect when chaining the three steps gnatprep && gnatchop && gnatmake,
> the compiler messages will in the end refer to the intermediate file
> that gnatprep will have created.

I haven't tried it but according to the GNAT User's Guide, if you pass
-r to both Gnatprep and Gnatchop, then compiler messages and debug
information will refer to the original file.

Björn Persson

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 12.02.13 17:04, Björn Persson wrote:
> Georg Bauhaus wrote:
>> As Björn said, the compiler should ideally be able to refer to the
>> original source file as usual. Now, while switch -m does have the desired
>> effect when chaining the three steps gnatprep && gnatchop && gnatmake,
>> the compiler messages will in the end refer to the intermediate file
>> that gnatprep will have created.
> 
> I haven't tried it but according to the GNAT User's Guide, if you pass
> -r to both Gnatprep and Gnatchop, then compiler messages and debug
> information will refer to the original file.

The process is actually spoiled at this stage, since -r will
either mean referring to an intermediate file (viz gnatprep's output),
or will not work because -gnatep leads to incorrect offsets.
I usually pass -r -w -c to gnatchop, which works fine except when
preprocessing is involved.

Maybe some insane trickery with pragma Source_File_Name
will work.

Re: chopping Ada source that have preprocessor symbols in them

[Stephen Leake]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> On 09.02.13 03:51, Stephen Leake wrote:
>> You have a file in CM:
>>
>> a_file.ada.gp
>>
>> It needs to be processed by gnatprep, then by gnatchop, and finally by
>> gprbuild.
>
> The problem is that running the chain of tools again and again is
> prohibitively inefficient.  It boils down to recompiling the world
> after each edit.
>
> The full picture is this: I am given, and will give back, a file,
>
> a_file.ada.gp
>
> It has about two dozen compilation units in it, some of them have many
> generic instantiations that will keep the compiler busy. I need to
> edit just some unit and then compile the program again.
>
> There are two "configurations", C1, and C2. Starting in C1,
> suppose I first run gnatprep on a_file.ada.gp, then gnatchop the
> preprocessed output. The two steps will produce about two dozen files.
> I can then call gnatmake as usual.

This is _precisely_ why the GNAT policy is one compilation unit per
file.

The root cause of your problem is putting more than one compilation unit
in one file. 

Change that.

Or get a faster computer; I hardly notice the time to compile a dozen
files.

-- 
-- Stephe

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 15.02.13 11:40, Stephen Leake wrote:
> The root cause of your problem is putting more than one compilation unit
> in one file.
>
> Change that.

What, twist Ada to meet GNAT requirements, or change the procedures
in the foreign organization?  :-) :-) :-)

Well, one unit per file *is* o.K, if *files* of a single piece of text
are any good (they are, absent a better alternative). Did you notice
that every IDE abstracts files away? (C-c C-d is among my favorites.)

> Or get a faster computer;

The old Ada problem again: Get a big budget...  My fairly recent
4 core machine should suffice for Ada, I hope, for the time
being. Alternatively, one could improve GNAT, -gnatep is there, it is
almost working, just not always, in particular with -m  ... ;-)

>  I hardly notice the time to compile a dozen
> files.

Compilation time changes when there are lots of generic instantiations
involved.

I'm moving parts from there to ordinary packages, which helps
in other ways, too.

Re: chopping Ada source that have preprocessor symbols in them

[Pascal Obry]

Le 15/02/2013 12:11, Georg Bauhaus a écrit :
> What, twist Ada to meet GNAT requirements, or change the procedures
> in the foreign organization?  :-) :-) :-)

Well AFAIK the actual representation of an Ada source is outside of the
Ada standard. So nothing to twist there :)

-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://v2p.fr.eu.org
  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

Re: chopping Ada source that have preprocessor symbols in them

[Randy Brukardt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

"Pascal Obry" <pascal@obry.net> wrote in message 
news:511E64AB.8030805@obry.net...
> Le 15/02/2013 12:11, Georg Bauhaus a �crit :
>> What, twist Ada to meet GNAT requirements, or change the procedures
>> in the foreign organization?  :-) :-) :-)
>
> Well AFAIK the actual representation of an Ada source is outside of the
> Ada standard. So nothing to twist there :)

That's not really true anymore for Ada 2012: compilers are required to take 
source files represented in UTF-8 with a direct mapping to the characters 
described in the Standard. And the Standard definitely describes multiple 
units in a file. Of course, exactly the steps needed to support those source 
files isn't described, and it could be long.

I sympathize with Georg's problem, but the point the Stephen made is 
well-taken: if modifying a file containing multiple units causes too much 
recompilation, then you have to split the file into several units. 
Certainly, I do that occassionally for Janus/Ada development -- usually 
things compile fast enough that it doesn't matter -- but if it starts to 
become annoying, I don't blame my tools working as designed, I blame my own 
sub-optimal choice of organization.

(And the use of a preprocessor is always suboptimal. It's much better to 
have that managed as part of version control, by selecting files 
specifically for particular targets -- but I've never seen a version control 
system that does that properly -- they all assume the use of preprocessors. 
For RRS, I built a front-end that adds such structure to the versioning (CVS 
is the back-end), but it's pretty clunky and never was turned into a 
product.)

                                                      Randy.

Re: chopping Ada source that have preprocessor symbols in them

[Stephen Leake]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> What, twist Ada to meet GNAT requirements, or change the procedures
> in the foreign organization?  :-) :-) :-)

I gather you believe changing the procedures in the foreign organization
is impossible; have you at least asked? Explained your problem and asked
for solutions?

> Well, one unit per file *is* o.K, if *files* of a single piece of text
> are any good (they are, absent a better alternative). Did you notice
> that every IDE abstracts files away? (C-c C-d is among my favorites.)

I don't see that as "abstracting files away"; C-c C-d (in Emacs Ada
mode) navigates to another file. Files are quite central to editing in
Emacs.

>> Or get a faster computer;
>
> The old Ada problem again: Get a big budget...  

It's hardly unique to Ada; recent Microsoft products make much bigger
demands on my computer, for far less gain.

> My fairly recent 4 core machine

The disk speed, memory size, and CPU clock rate (in that order) have
more impact; the gnat compiler does not take advantage of multiple cores
(unless I've missed a recent change), and compiling does a lot of disk IO.

> Alternatively, one could improve GNAT, -gnatep is there, it is almost
> working, just not always, in particular with -m ... ;-)

Right. Sounds like a reasonable enhancement request.

>>  I hardly notice the time to compile a dozen
>> files.
>
> Compilation time changes when there are lots of generic instantiations
> involved.

Yes, the details matter. Improving generic instantiation compilation
speed might also be a reasonable enhancement request; AdaCore can always
use more feedback on what to work on next.

-- 
-- Stephe

Re: chopping Ada source that have preprocessor symbols in them

[Georg Bauhaus]

On 16.02.13 09:22, Stephen Leake wrote:

> I gather you believe changing the procedures in the foreign organization
> is impossible; have you at least asked? Explained your problem and asked
> for solutions?

I have asked for why this arrangement of source code is needed; others
have asked before me. The reasons given are in fact convincing once
the context becomes apparent to the outsider.

> I don't see that as "abstracting files away"; C-c C-d (in Emacs Ada
> mode) navigates to another file. Files are quite central to editing in
> Emacs.

The programmer using an IDE does not have to remember the files, the
file naming scheme, etc.(*) Source code navigation and language-based
views give a better outline of the program's structure. Files are
central to how Emacs and other IDEs fill and save text buffers, sure,
but the files vanish when editing with good language support. "Get me
the declaration of that thing!" If this is what the programmers wish
the IDE to do for them, then why should they have to do all the name
resolution themselves? I.e. find full Ada names and then know the
corresponding file system names?

>> My fairly recent 4 core machine
>
> The disk speed, memory size, and CPU clock rate (in that order) have
> more impact; the gnat compiler does not take advantage of multiple cores
> (unless I've missed a recent change), and compiling does a lot of disk IO.

File I/O is not slowing the process much; what seems to matter is the
dependency mechanism that GNAT is using (which, I think, is normally
stat(2)-based).  This would be the reason why GNAT's switch -m helps
so much whenever it works with freshly chopped source.


> Yes, the details matter. Improving generic instantiation compilation
> speed might also be a reasonable enhancement request; AdaCore can always
> use more feedback on what to work on next.

In case of generic instantiations, I'd actually expect, or even hope
that compilation time increases. When the compiler spends time
handling instances this could reflect efforts to adapt instances to
the specifics of the generic formals.  Indeed, the quality of the code
produced lets me think that compilation time was well spent. I have
few compilers to compare, but for example, I see what happens to
statically known conditionals whose value depends on generic
formals---static polymorphism at its best, without the need for a
tagged dummy variable. This counts when inner loops perform no
comparison instead of an average of 2, so waiting for the compiler is
worth it. (That's assuming that the program would not work better
if all instances shared the same code of a smaller executable.)

__
(*) Maybe this is a reason why some programmers coming from Turbo
Pascal-like IDEs would have to learn about files, editors, command
line compilers etc. They had been developing software without these
artifacts of OS bureaucracy.

Re: chopping Ada source that have preprocessor symbols in them

[Jacob Sparre Andersen]

Randy Brukardt wrote:

> (And the use of a preprocessor is always suboptimal. It's much better
> to have that managed as part of version control, by selecting files
> specifically for particular targets -- but I've never seen a version
> control system that does that properly -- they all assume the use of
> preprocessors.  For RRS, I built a front-end that adds such structure
> to the versioning (CVS is the back-end), but it's pretty clunky and
> never was turned into a product.)

How would you do that in version control?

Personally I like the style used with the GNAT project files, where you
select different implementation variants with a parameter to the build
system.

I guess you would like to be able to select a "branch" with a specific
implementation variant _and_ at the same time select a (different)
"branch" of all the invariant parts of the system.

Now that I think about it, you could do that by having a subrepository
containing the implementation variants in different branches.

Greetings,

Jacob
-- 
"Reality is that which, when you stop believing in it, 
 doesn't go away."

Re: chopping Ada source that have preprocessor symbols in them

[Stephen Leake]

Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:

> On 16.02.13 09:22, Stephen Leake wrote:
>
>> I gather you believe changing the procedures in the foreign organization
>> is impossible; have you at least asked? Explained your problem and asked
>> for solutions?
>
> I have asked for why this arrangement of source code is needed; others
> have asked before me. The reasons given are in fact convincing once
> the context becomes apparent to the outsider.

Ok.

>> I don't see that as "abstracting files away"; C-c C-d (in Emacs Ada
>> mode) navigates to another file. Files are quite central to editing in
>> Emacs.
>
> The programmer using an IDE does not have to remember the files, the
> file naming scheme, etc.(*) Source code navigation and language-based
> views give a better outline of the program's structure. 

Maybe for languages other than Ada; Ada files map quite nicely to Ada
program structure (assuming one compilation unit per file).

> Files are central to how Emacs and other IDEs fill and save text
> buffers, sure, but the files vanish when editing with good language
> support. 

They don't vanish completely. You still have to find the first Ada file
you are interested in! And the related test code, and the package that
contains the derived type (although help may be comming for that in
Emacs Ada mode 5.1 :), and the LaTeX file that contains the
documentation, and the .gpr file that contains the compilation options,
and the Makefile. Etc.

> "Get me the declaration of that thing!" If this is what the
> programmers wish the IDE to do for them, then why should they have to
> do all the name resolution themselves? 

Yes, an IDE should _help_ with file navigation when it can; that does
not mean file navigation vanishes _completely_.

-- 
-- Stephe

Re: chopping Ada source that have preprocessor symbols in them

[Björn Persson]

Stephen Leake wrote:
> Georg Bauhaus <rm.dash-bauhaus@futureapps.de> writes:
> > My fairly recent 4 core machine  
> 
> The disk speed, memory size, and CPU clock rate (in that order) have
> more impact; the gnat compiler does not take advantage of multiple cores
> (unless I've missed a recent change), and compiling does a lot of disk IO.

Gnatmake and GPRbuild will compile multiple files in parallel if you
use the -j switch, and that's not exactly new.

Björn Persson

Re: chopping Ada source that have preprocessor symbols in them

[Randy Brukardt]

"Jacob Sparre Andersen" <sparre@nbi.dk> wrote in message 
news:871ucffhtu.fsf@adaheads.sparre-andersen.dk...
> Randy Brukardt wrote:
>
>> (And the use of a preprocessor is always suboptimal. It's much better
>> to have that managed as part of version control, by selecting files
>> specifically for particular targets -- but I've never seen a version
>> control system that does that properly -- they all assume the use of
>> preprocessors.  For RRS, I built a front-end that adds such structure
>> to the versioning (CVS is the back-end), but it's pretty clunky and
>> never was turned into a product.)
>
> How would you do that in version control?

You keep the relationships between related-but-different files. I don't know 
of any version control that can really do that.

Also, you ought to be able to extract an entire buildable product from the 
version control with a single command; if you can do that (and have an 
appropriate build mechanism), you don't really need any other management 
technique (i.e. GPR files).

> Personally I like the style used with the GNAT project files, where you
> select different implementation variants with a parameter to the build
> system.

That covers the problem of finding the varied source files, but doesn't 
solve the problem of keeping them in synch. The former problem is easy, the 
latter is hard (and critical).

> I guess you would like to be able to select a "branch" with a specific
> implementation variant _and_ at the same time select a (different)
> "branch" of all the invariant parts of the system.
>
> Now that I think about it, you could do that by having a subrepository
> containing the implementation variants in different branches.

Branching (as I've seen it implemented) really does not help.

There are three kinds of files in a typical system. Consider the Janus/Ada 
compiler and a modernized example of reality. There needs to be code 
generated for Intel 32-bit and 64-bit targets, as well as ARM targets. And 
Windows and Linux.

Most of the files of the compiler don't depend on the host or target at all, 
or only depend on them indirectly through descriptor packages.

Some files (like part of an ARM code generator) don't appear in any other 
compiler version at all. So a mechanism like the GPR files would work fine 
for those.

But other files have varied versions for different targets. The "weak" 
version of that is where the specification is shared and the bodies are 
unrelated. GPRs will help you with that, but you have a problem if you need 
to add a new item to the specification -- how do you find all of the bodies 
that need changes?

But the real situation is that you have different but related versions for 
different targets. For instance, the host descriptor packages contain a 
large bunch of constants, some of which vary by target. What you don't want 
to do is have completely separate copies of these packages, because it makes 
changing one of the packages to add or delete something difficult -- now you 
have to change a bunch of other packages as well.

Moreover, those bodies that are different for a shared body rarely are 100% 
custom for each target. Typically, the algorithms are similar in each target 
(and certainly the declarations are), and these shared parts really ought to 
be shared (so that if a modification needs to be made in one to fix a bug, 
it will get fixed in all versions).

I suppose one could build something on top of an existing version control 
system (after all, this is what we tried to do back in the early 1990s). And 
it probably would have resemblance to a reversible preprocessor. Probably it 
would have to be built into an IDE. The trick would be for the IDE to manage 
the branching and making the changes to the appropriate parts of the source 
file so that the common parts get updated for all versions and the not 
common parts are only changed in one version.

The problem is figuring out an appropriate user-interface for this; I never 
cracked that nut, and the whole idea doesn't work if it can't be done so 
people understand it. (Otherwise it just is another only-for-me program.)

                                             Randy.

Version control and multiple implementation variants (Was: chopping Ada source that have preprocessor symbols in them)

[Jacob Sparre Andersen]

Randy Brukardt wrote:
> Jacob Sparre Andersen wrote:
>> Randy Brukardt wrote:

> Also, you ought to be able to extract an entire buildable product from
> the version control with a single command; if you can do that (and
> have an appropriate build mechanism), you don't really need any other
> management technique (i.e. GPR files).

True.

I've looked a little bit at Mercurial and its "subrepos" facility
(http://mercurial.selenic.com/wiki/Subrepository).  I think it (combined
with a sensible use of separate bodies) does the trick.

Taking Janus/Ada with two OS targets and two CPU architectures, you
would set up a structure like this:

   Janus_Ada/ - thin build management repository
      Common/       - subrepo containing source files common to all targets
      OS_Specific/  - subrepo containing branches for Linux/Windows
      CPU_Specific/ - subrepo containing branches for ARM/Intel

When you commit in the thin master repository, it will register which
revision each of the subrepos are in, such that when you later check
that revision out, the appropriate revisions will also be checked out in
the subrepos.

Commits in the master repository are aborted when there are uncommitted
changes in subrepos.  This will help assuring that a test-build actually
matches what is committed in the build management repository.

(Warning: Not tested.)

> Branching (as I've seen it implemented) really does not help.
>
> There are three kinds of files in a typical system. Consider the Janus/Ada 
> compiler and a modernized example of reality. There needs to be code 
> generated for Intel 32-bit and 64-bit targets, as well as ARM targets. And 
> Windows and Linux.
>
> Most of the files of the compiler don't depend on the host or target
> at all, or only depend on them indirectly through descriptor packages.

These files are the ones I would put in "Common" in the example above.

> Some files (like part of an ARM code generator) don't appear in any
> other compiler version at all. So a mechanism like the GPR files would
> work fine for those.

Yes.  In the example above, I would commit these files in the "ARM"
branch of "CPU_Specific".

> But other files have varied versions for different targets. The "weak" 
> version of that is where the specification is shared and the bodies are 
> unrelated. GPRs will help you with that, but you have a problem if you need 
> to add a new item to the specification -- how do you find all of the bodies 
> that need changes?

I think I would check the GPR "Naming" package for the list of implementation
variants.

In the example above I would have to check out (select) each branch in
the relevant subrepo to see and change that body.

> But the real situation is that you have different but related versions
> for different targets. For instance, the host descriptor packages
> contain a large bunch of constants, some of which vary by target. What
> you don't want to do is have completely separate copies of these
> packages, because it makes changing one of the packages to add or
> delete something difficult -- now you have to change a bunch of other
> packages as well.

Wouldn't this be doable with nested packages with separate bodies for
the target specific parts?

> Moreover, those bodies that are different for a shared body rarely are
> 100% custom for each target. Typically, the algorithms are similar in
> each target (and certainly the declarations are), and these shared
> parts really ought to be shared (so that if a modification needs to be
> made in one to fix a bug, it will get fixed in all versions).

Yes.  I think this is an example where it would make sense to use nested
packages with separate bodies for the target specific parts.  (But I
have never written an Ada compiler, so I may be wrong.)

> I suppose one could build something on top of an existing version
> control system (after all, this is what we tried to do back in the
> early 1990s). And it probably would have resemblance to a reversible
> preprocessor. Probably it would have to be built into an IDE. The
> trick would be for the IDE to manage the branching and making the
> changes to the appropriate parts of the source file so that the common
> parts get updated for all versions and the not common parts are only
> changed in one version.

With Mercurial and subrepos the IDE can basically run "hg branches" in
each subrepo to get a list of possible settings, turning it into a
drop-down list in the user interface.

> The problem is figuring out an appropriate user-interface for this; I
> never cracked that nut, and the whole idea doesn't work if it can't be
> done so people understand it. (Otherwise it just is another
> only-for-me program.)

I hope the ideas above are useful.

Greetings,

Jacob

PS: When will you start accepting pre-orders for a Linux version of
    Janus/Ada?
-- 
"In school, and in most aspects of life, a 90% is an A.
 In software, a 99.9% is, or may be, an utter disaster."

Re: Version control and multiple implementation variants (Was: chopping Ada source that have preprocessor symbols in them)

[Thomas Løcke]

On 02/19/2013 11:05 AM, Jacob Sparre Andersen wrote:
> PS: When will you start accepting pre-orders for a Linux version of
>      Janus/Ada?


Oooh, I'm intrigued by this question. Janus/Ada for Linux sounds like a
very nice thing. I wonder what the pricerange would be?  :)


-- 
Thomas L�cke | thomas@12boo.net | http://12boo.net

Re: chopping Ada source that have preprocessor symbols in them

[Pascal Obry]

Le 19/02/2013 11:05, Jacob Sparre Andersen a �crit :
> I've looked a little bit at Mercurial and its "subrepos" facility
> (http://mercurial.selenic.com/wiki/Subrepository).  I think it (combined
> with a sensible use of separate bodies) does the trick.
> 
> Taking Janus/Ada with two OS targets and two CPU architectures, you
> would set up a structure like this:
> 
>    Janus_Ada/ - thin build management repository
>       Common/       - subrepo containing source files common to all targets
>       OS_Specific/  - subrepo containing branches for Linux/Windows
>       CPU_Specific/ - subrepo containing branches for ARM/Intel

But this means that to do a fix for multiple OS/CPU (for example adding
a new routine) you have to move branches! This is quite counter
productive and it also means that you cannot do an atomic commit for
related work on different branches.

I do prefer the GNAT way. That is, having files for different OS/CPU on
the same repo with a suffix starting with double _ to avoid having them
picked up by default by the project file (as this is not a standard
naming). Then using the package Naming and some variables you select the
variant you want to use for a specific build.

-- 
  Pascal Obry /  Magny Les Hameaux (78)

  The best way to travel is by means of imagination

  http://v2p.fr.eu.org
  http://www.obry.net

  gpg --keyserver keys.gnupg.net --recv-key F949BD3B

Re: Version control and multiple implementation variants (Was: chopping Ada source that have preprocessor symbols in them)

[Randy Brukardt]

"Jacob Sparre Andersen" <sparre@nbi.dk> wrote in message 
news:87liakk4aa.fsf_-_@adaheads.sparre-andersen.dk...
> Randy Brukardt wrote:
...
>> But the real situation is that you have different but related versions
>> for different targets. For instance, the host descriptor packages
>> contain a large bunch of constants, some of which vary by target. What
>> you don't want to do is have completely separate copies of these
>> packages, because it makes changing one of the packages to add or
>> delete something difficult -- now you have to change a bunch of other
>> packages as well.
>
> Wouldn't this be doable with nested packages with separate bodies for
> the target specific parts?

Not really, because Ada requires some things to be static (the size in bits 
for type Integer, for example), and thus they have to be defined as 
constants in a package specification.

Some such things (that don't have to be static) can be described as 
functions and thus put into a body, but when you do that you're making your 
program slower and more complex (especially as memory management is 
concerned; arrays with compile-time bounds are directly allocated while 
other arrays have to be allocated using some dynamic mechanism). Obviously, 
this matters for a compiler.

 >> Moreover, those bodies that are different for a shared body rarely are
>> 100% custom for each target. Typically, the algorithms are similar in
>> each target (and certainly the declarations are), and these shared
>> parts really ought to be shared (so that if a modification needs to be
>> made in one to fix a bug, it will get fixed in all versions).
>
> Yes.  I think this is an example where it would make sense to use nested
> packages with separate bodies for the target specific parts.  (But I
> have never written an Ada compiler, so I may be wrong.)

My level of sharing is much more fine-grained than yours. I'm thinking about 
any individual line of text that's shared, like the declarations of the 
various subprograms. For instance, in something like:

    function Register_Possible (Start, Stop : Int_Code_Index; Item : 
Int_Code_Index) return Boolean is
         -- Returns True if Item can be placed into a target hardware 
register over the
         -- entire range of code represented by Start .. Stop.
         Max_Registers : constant := 6; -- Intel x86.
    begin
         for I in Start .. Stop loop
               -- Figure the likely number of registers used by this code, 
return false if it ever reaches Max_Registers.
              case Code(I).Op_Code is
                   when RAISE_OP | TKCREATE | ... =>
                          -- These kill (use) all registers, so the answer 
is no.
                          return False;
                   ... -- Other intermediate code instructions.
              end case;
        end loop;
        return True; -- If we get here, we can put this item into a 
register.
    end Register_Possible;

In this code, the only code that's actually target-specific is the constant 
Max_Registers and the code represented by the ... The part about 
instructions that kill all registers is not target specific (obviously). (It 
should be noted that we use case statements for this sort of code as much as 
possible, without others clauses, because they let the Ada compiler complain 
if we forget to add a new instruction to the intermediate form, something 
that isn't common but does happen.)

Because so much of these algorithms is not target-specific, and the use of 
case statements is so important for correctness, I want to be able to manage 
these files as related (so that changes to the common parts get made 
automatically) and the unshared parts get managed separately.

(Side-note: Ada 2012's static predicates will allow a bit of additional 
abstraction here, as we will be able to define a subtype to represent all 
register-killing instructions and the like. That will at least eliminate one 
common thing that has to be maintained in all of these versions. But of 
course most of the code is in the limb, not the list of choices.)

...
> PS: When will you start accepting pre-orders for a Linux version of
>    Janus/Ada?

I'm happy to take people's money at any time. However, no one should be 
expecting delivery any time soon. ;-)

                                                     Randy.

Re: Version control and multiple implementation variants (Was: chopping Ada source that have preprocessor symbols in them)

[Georg Bauhaus]

On 19.02.13 22:07, Randy Brukardt wrote:
> My level of sharing is much more fine-grained than yours. I'm thinking about
> any individual line of text that's shared,

A similar situation that requires very small changes only is when a
software component needs to be compiled for both Ada 95 and Ada
2005. It works in many cases, but there are differences between the
versions of the language that won't allow the same source to be
accepted by both compilers.

When that is the case, there will be small differences, and they will
stay even when the language version does not affect most of the
"algorithmic lines" of the source text.  I imagine that using version
control and branches, I will be injecting lots of submit and merge
steps between the usual edit and compile steps. Sounds suboptimal.