Ann: Gcc/GNAT 9.1 released - Test Results

Ann: Gcc/GNAT 9.1 released - Test Results

[alby.gamper]

Dear Ada Community

GCC 9.1 has been released, and below are the test results for GNAT (I will try and
run the ACATS version 4.1 using Simon Wright's GitHub repo at a later stage)



                === acats configuration ===
target gcc is /home/alexg/Build/gcc/xgcc -B/home/alexg/Build/gcc/
 Configured with: ../Software/gcc/configure --prefix=/mingw64 --with-sysroot=/mingw64 --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --disable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=ada,c,lto,c++ --enable-shared --enable-static --enable-libatomic --enable-threads=win32 --enable-graphite --enable-fully-dynamic-string --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr gcc version 9.1.1 20190504 [ibm/gcc-9-branch revision 270862]' --with-gnu-as --with-gnu-ld
host=x86_64-w64-mingw32
target=x86_64-w64-mingw32
gnatmake is /home/alexg/Build/gcc/gnatmake

                === acats tests ===
Running chapter a ...
Running chapter c2 ...
FAIL:   c23003b
FAIL:   c23003g
FAIL:   c23003i
FAIL:   c250002
Running chapter c3 ...
FAIL:   c380004
Running chapter c4 ...
Running chapter c5 ...
Running chapter c6 ...
Running chapter c7 ...
Running chapter c8 ...
Running chapter c9 ...
Running chapter ca ...
Running chapter cb ...
Running chapter cc ...
Running chapter cd ...
Running chapter ce ...
Running chapter cxa ...
Running chapter cxb ...
Running chapter cxf ...
Running chapter cxg ...
Running chapter cxh ...
Running chapter cz ...
Running chapter d ...
Running chapter e ...
Running chapter l ...
                === acats Summary ===
# of expected passes            2315
# of unexpected failures        5
*** FAILURES: c23003b c23003g c23003i c250002 c380004

                === gnat tests ===
FAIL: gnat.dg/entry_queues.adb execution test
                === gnat Summary ===

# of expected passes            2965
# of unexpected failures        1
# of expected failures          23
# of unsupported tests          9

/home/alexg/Build/gcc/gnatmake version 9.1.1 20190504 [ibm/gcc-9-branch revision 270862]

Alex

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[Simon Wright]

alby.gamper@gmail.com writes:

> GCC 9.1 has been released, and below are the test results for GNAT (I
> will try and run the ACATS version 4.1 using Simon Wright's GitHub
> repo at a later stage)

With release GCCs, c452003 will hang during compilation (the test suite
checks for overruns of executions, but not of builds). Killing the hung
gnat1 process lets ACATS proceed.

With non-release GCCs, which have many more checks enabled, you get a
bug box: see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88610

For info, the GCC 9.1.0 x86_64-apple-darwin official results are:

		=== gnat Summary ===

# of expected passes		2964
# of expected failures		23
# of unsupported tests		10

		=== acats Summary ===
# of expected passes		2320
# of unexpected failures	0

The 4.1L results are

		=== acats Summary ===
# of expected passes		2505
# of unexpected failures	11
# of expected failures		1463
# of unresolved testcases	11
# of unsupported tests		124
*** FAILURES: c250002 c452003 c611a04 c650b04 cxd1003 cxd1004 cxd1005 cxd2006 cxd3001 cxd3002 cxd8002 

There are expected failures because I included the whole test suite.

c250002: UTF-8 case conversion issue
c452003: see above
c6*:     hairy tests
cxd*:    Bob Duff says full Annex D compliance not to be expected on
         desktop OS

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[Optikos]

On Wednesday, May 8, 2019 at 12:09:13 PM UTC-5, Simon Wright wrote:
> alby.gamper@gmail.com writes:
> 
> > GCC 9.1 has been released, and below are the test results for GNAT (I
> > will try and run the ACATS version 4.1 using Simon Wright's GitHub
> > repo at a later stage)
> 
> With release GCCs, c452003 will hang during compilation (the test suite
> checks for overruns of executions, but not of builds). Killing the hung
> gnat1 process lets ACATS proceed.
> 
> With non-release GCCs, which have many more checks enabled, you get a
> bug box: see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88610
> 
> For info, the GCC 9.1.0 x86_64-apple-darwin official results are:
> 
> 		=== gnat Summary ===
> 
> # of expected passes		2964
> # of expected failures		23
> # of unsupported tests		10
> 
> 		=== acats Summary ===
> # of expected passes		2320
> # of unexpected failures	0
> 
> The 4.1L results are
> 
> 		=== acats Summary ===
> # of expected passes		2505
> # of unexpected failures	11
> # of expected failures		1463
> # of unresolved testcases	11
> # of unsupported tests		124
> *** FAILURES: c250002 c452003 c611a04 c650b04 cxd1003 cxd1004 cxd1005 cxd2006 cxd3001 cxd3002 cxd8002 
> 
> There are expected failures because I included the whole test suite.

Ignoring MacOS and perhaps all the BSDs for a moment, does Bob Duff's pardon extend to Linux kernels that are configured with the full set of realtime extensions (regardless of being deployed on a GUI-based desktop by happenstance)?  It seems that Linux with the full set of realtime extensions turned on would be expected to fully meet Annex D.

> c250002: UTF-8 case conversion issue
> c452003: see above
> c6*:     hairy tests
> cxd*:    Bob Duff says full Annex D compliance not to be expected on
>          desktop OS

Which of these unsupported tests and unexpected failures (and unexpected successses*, if any arise) are •not• each tracked in GCC's Bugzilla with their own individual defect report?

* Ada source code that the ARM (or AARM's additional color) deems impermissible Ada source code that GNAT failed to prohibit via a (nonbugbox) compilation error.

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[Simon Wright]

Optikos <optikos@verizon.net> writes:

>> For info, the GCC 9.1.0 x86_64-apple-darwin official results are:
>>
>> 		=== gnat Summary ===
>>
>> # of expected passes		2964
>> # of expected failures		23
>> # of unsupported tests		10
>>
>> 		=== acats Summary ===
>> # of expected passes		2320
>> # of unexpected failures	0
>>
>> The 4.1L results are
>>
>> 		=== acats Summary ===
>> # of expected passes		2505
>> # of unexpected failures	11
>> # of expected failures		1463
>> # of unresolved testcases	11
>> # of unsupported tests		124
>> *** FAILURES: c250002 c452003 c611a04 c650b04 cxd1003 cxd1004
>> cxd1005 cxd2006 cxd3001 cxd3002 cxd8002
>>
>> There are expected failures because I included the whole test suite.
>
> Ignoring MacOS and perhaps all the BSDs for a moment, does Bob Duff's
> pardon extend to Linux kernels that are configured with the full set
> of realtime extensions (regardless of being deployed on a GUI-based
> desktop by happenstance)?  It seems that Linux with the full set of
> realtime extensions turned on would be expected to fully meet Annex D.
>
>> c250002: UTF-8 case conversion issue
>> c452003: see above
>> c6*:     hairy tests
>> cxd*:    Bob Duff says full Annex D compliance not to be expected on
>>          desktop OS

I don't think I have full evidence for the opinion I ascribed to Bob
Duff. The reference I have is at [1]:

   "Ceiling locking is supported on Linux, but that support is fairly
   recent (I implemented it in the last few months or so).  I think GNAT
   GPL 2016 doesn't have it."

   "You have to link with the right library, and you have to set the
   appropriate capability on the executable file (which requires being
   root), and then you can run that file without being root and get
   ceiling priority support."

See also [2] (this documentation is for the supported GNAT; CE and FSF
releases may well be behind).

[1] https://groups.google.com/d/msg/comp.lang.ada/aKIP5L54kvU/f15BSLn2AAAJ
[2]
https://docs.adacore.com/gnat_rm-docs/html/gnat_rm/gnat_rm/implementation_of_specific_ada_features.html#gnat-implementation-of-tasking

> Which of these unsupported tests and unexpected failures (and
> unexpected successses*, if any arise) are •not• each tracked in GCC's
> Bugzilla with their own individual defect report?
>
> * Ada source code that the ARM (or AARM's additional color) deems
> impermissible Ada source code that GNAT failed to prohibit via a
> (nonbugbox) compilation error.

There are presently no unexpected passes.

Quite a few of the unsupported tests are so because they deal with the
consequences of changing the source code and rebuilding. The ACATS
source for those tests contains multiple copies of some units, which
results in a gnatchop failure. It'd require a lot of work to fix, and
for what is essentially an IVP (installation verification procedure)
seems like overkill.

Some are unsupported because the test determines for itself that it is
inappropriate (e.g. handling of floating-point overflow).

Failures: I'm sure that AdaCore will eventually deal with the new
unexpected failures, but I'm not sure that posting PRs will help:
viz. response to PR 88610,

   "My point is that opening a PR has strictly no effects in this case,
   unlike for other cases. New ACATS tests are publicly available and
   the compiler will be changed accordingly at some point, but the
   priority is very low."

I still think that ICEs (internal compiler errors, => bug box) are
different in kind from failure to compile or failure to execute
properly. As I said,

   "I’d certainly think twice before reporting a mere compilation
   failure in what’s a pretty obscure language area. But this is an
   ICE."

CXD: there are failures in this area on Linux, but these are on macOS,
and there's not a huge customer market there! Also, would you think it
appropriate to base a hard real-time system on macOS? I wouldn't.

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[alby.gamper]

On Thursday, May 9, 2019 at 3:09:13 AM UTC+10, Simon Wright wrote:
> alby.gamper@gmail.com writes:
> 
> > GCC 9.1 has been released, and below are the test results for GNAT (I
> > will try and run the ACATS version 4.1 using Simon Wright's GitHub
> > repo at a later stage)
> 
> With release GCCs, c452003 will hang during compilation (the test suite
> checks for overruns of executions, but not of builds). Killing the hung
> gnat1 process lets ACATS proceed.
> 
> With non-release GCCs, which have many more checks enabled, you get a
> bug box: see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88610
> 
> For info, the GCC 9.1.0 x86_64-apple-darwin official results are:
> 
> 		=== gnat Summary ===
> 
> # of expected passes		2964
> # of expected failures		23
> # of unsupported tests		10
> 
> 		=== acats Summary ===
> # of expected passes		2320
> # of unexpected failures	0
> 
> The 4.1L results are
> 
> 		=== acats Summary ===
> # of expected passes		2505
> # of unexpected failures	11
> # of expected failures		1463
> # of unresolved testcases	11
> # of unsupported tests		124
> *** FAILURES: c250002 c452003 c611a04 c650b04 cxd1003 cxd1004 cxd1005 cxd2006 cxd3001 cxd3002 cxd8002 
> 
> There are expected failures because I included the whole test suite.
> 
> c250002: UTF-8 case conversion issue
> c452003: see above
> c6*:     hairy tests
> cxd*:    Bob Duff says full Annex D compliance not to be expected on
>          desktop OS

Hi Simon

I recall having a issue with one of the tests (and I think it was also c452003)
where I had to kill the hung compilation. And this was with FSF 8.3 running
your ACATS 4.1 repo. When I get a chance I'll rerun the ACATS 4.1 test on FSF 9.1

Alex

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[Simon Wright]

alby.gamper@gmail.com writes:

> I recall having a issue with one of the tests (and I think it was also
> c452003) where I had to kill the hung compilation. And this was with
> FSF 8.3 running your ACATS 4.1 repo. When I get a chance I'll rerun
> the ACATS 4.1 test on FSF 9.1

I'd expect a compilation failure (which this surely is!) to fail the
same on all OSs, in most cases anyway.

Runtime failures (e.g. the CXDs) are of course different.

I was interested to see you had a c250002 failure: I expected this one
to fail on Windows as it does on macOS, see PR 81114, GNAT mishandles
filenames with UTF8 chars on case-insensitive filesystems.
                   
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81114

Re: Ann: Gcc/GNAT 9.1 released - Test Results

[Randy Brukardt]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lyk1f0vw11.fsf@pushface.org...
...
> Quite a few of the unsupported tests are so because they deal with the
> consequences of changing the source code and rebuilding. The ACATS
> source for those tests contains multiple copies of some units, which
> results in a gnatchop failure. It'd require a lot of work to fix, and
> for what is essentially an IVP (installation verification procedure)
> seems like overkill.

Note that the (relatively new) ACATS grading tool checks process failures, 
and the GNAT scripting tool available in the submitted tests area handles 
most of these tests properly.

I note that my experience is that there are a lot of B-Tests (on *every* 
compiler that I've checked) that are either in a grey area as to whether 
they have passed or are outright failing. I suspect that's because errors 
creep into supposedly "known good results", and also because without any 
formal verifications it is really easy to decide not to worry about a 
dubious result. (It's also possible that my test procedure didn't exactly 
match the one typically used by the vendor, so some results might differ).

                                         Randy.