FYI - ACATS test results for GCC 8.1.1 (mingw)

FYI - ACATS test results for GCC 8.1.1 (mingw)

[alby.gamper]

Thanks to the effort of Simon Wright's on his GitHub project, With very little
tweaking, I was able to run the ACATS's test on my mingw64 environment, with
what I believe is a very successful result !

Please note however, that this is a slightly modified build script of GCC as
supplied on GitHub (specifically the --disable-bootstrap option)

$ ~/ACATS/run_local.sh
Test Run By alex.gamper on 20 Jun 2018 16:34:46
                === acats configuration ===
target gcc is /mingw64/bin/gcc
 Configured with: ../gcc/configure
--prefix=/mingw64 --with-local-prefix=/mingw64/local
--build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
--with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include
--libexecdir=/mingw64/lib
--disable-bootstrap 
--with-arch=x86-64 --with-tune=generic
--enable-languages=c,lto,c++,objc,obj-c++,fortran,ada
--enable-shared --enable-static
--enable-libatomic
--enable-threads=posix
--enable-graphite
--enable-fully-dynamic-string
--enable-libstdcxx-time=yes
--disable-libstdcxx-pch
--disable-libstdcxx-debug
--disable-isl-version-check
--enable-lto
--enable-libgomp
--disable-multilib
--enable-checking=release
--disable-rpath
--disable-win32-registry
--disable-nls
--disable-werror
--disable-symvers
--with-libiconv
--with-system-zlib
--with-gmp=/mingw64
--with-mpfr=/mingw64
--with-mpc=/mingw64
--with-isl=/mingw64
--with-pkgversion='Rev1, Built by MSYS2 project'
--with-bugurl=https://sour gcc version 8.1.1 20180619 (Rev1, Built by MSYS2 project)
host=x86_64-w64-mingw32
target=x86_64-w64-mingw32
gnatmake is /mingw64/bin/gnatmake

                === acats support ===
Generating support files... done.
Compiling support files... done.

                === acats Summary ===
# of expected passes            2504
# of unexpected failures        8
# of expected failures          1458
# of unresolved testcases       11
# of unsupported tests          124
*** FAILURES: c250002 c380004 c52103x c52104x c611a04 c760a02 c760a03 cxag003

Thanks

Alex

Re: FYI - ACATS test results for GCC 8.1.1 (mingw)

[Simon Wright]

alby.gamper@gmail.com writes:

> Thanks to the effort of Simon Wright's on his GitHub project, With
> very little tweaking, I was able to run the ACATS's test on my mingw64
> environment, with what I believe is a very successful result !

Great!

>                 === acats support ===
> Generating support files... done.
> Compiling support files... done.
>
>                 === acats Summary ===
> # of expected passes            2504
> # of unexpected failures        8
> # of expected failures          1458
> # of unresolved testcases       11
> # of unsupported tests          124
> *** FAILURES: c250002 c380004 c52103x c52104x c611a04 c760a02 c760a03
> cxag003

On macOS, with 8.1.0,

		=== acats Summary ===
# of expected passes		2501
# of unexpected failures	11
# of expected failures		1458
# of unresolved testcases	11
# of unsupported tests		124
*** FAILURES: c250002 c611a04 c760a02 c760a03 cxag003 cxd1003 cxd1004 cxd1005 cxd2006 cxd3001 cxd3002 

Comments:

c250002: GNAT fails to take account of case-preserving UTF8 filesystems

c380004: discriminants & protected, task types

c52103x: (large) arrays

c52104x: likewise

c611a04: class-wide preconditions

c760a02: function call intializing immutably limited type

c760a03: likewise

cxag003: Name_Case_Kind, _Equivalence not in Ada.Directories

cxd1003: rendezvous inherits active priority of caller

cxd1004: task inherits the active priority of its parent

cxd1005: likewise

cxd2006: priority inherited after it should be

cxd3001: ceiling locking

cxd3002: likewise

========================================================================

Note that GCC regards darwin and mingw32 as secondary platforms, and the
release criteria <https://gcc.gnu.org/gcc-8/criteria.html> are

  * The compiler bootstraps successfully, and the C++ runtime library
    builds.
    
  * The DejaGNU testsuite has been run, and a substantial majority of
    the tests pass.

so I need to run the suite on a *86* linux box.

Note that "unsupported" means (in this suite) that the test was either
deemed not applicable by the test itself (for example, C45322A requires
Machine_Overflows to be True), not supported by the compiler (for
example, CXAG002 will not compile because it requires support for
Hierarchical_File_Names), or not compatible with simple use of gnatchop
(because the test contains the same unit in several source files,
expecting them to be used in multiple passes).

Note that "expected failures" includes the B tests, which are
expected to fail, and the L tests, which may fail. This suite does not
check that the test fails in the expected way.

Note that I should maybe categorise some of the tests as XFAIL (expected
to fail), e.g. cxag003, so that when the bug is corrected it will show
up as an XPASS (wasn't expected to pass). I can't do this with most of
the others, since the pass/fail seems to be machine-dependent. I'm
reluctant to not even attempt them!

I'd like to try some of the tasking-related tests on a Ravenscar target,
but it's quite a lot of work (easier on one of the AdaCore -full- RTSs,
because they support exception propagation).

Re: FYI - ACATS test results for GCC 8.1.1 (mingw)

[Dan'l Miller]

On Wednesday, June 20, 2018 at 8:16:53 AM UTC-5, Simon Wright wrote:
> Note that I should maybe categorise some of the tests as XFAIL (expected
> to fail), e.g. cxag003, so that when the bug is corrected it will show
> up as an XPASS (wasn't expected to pass).

No, please don't do that.  That would be incredibly confusing.  A long-term bug is still a bug.  (A deviance from the _LRM_ is still a bug, even if Robert Dewar coded it himself in 1993 and GNAT has stated officially that the sun will burn out before they fix it.)  Just because we expect a bug to be there long-term, doesn't mean that that test-case should cover up the bug to cook the books to make it look good.

Re: FYI - ACATS test results for GCC 8.1.1 (mingw)

[Simon Wright]

Simon Wright <simon@pushface.org> writes:

> Note that GCC regards darwin and mingw32 as secondary platforms, and
> the release criteria <https://gcc.gnu.org/gcc-8/criteria.html> are
>
>   * The compiler bootstraps successfully, and the C++ runtime library
>     builds.
>     
>   * The DejaGNU testsuite has been run, and a substantial majority of
>     the tests pass.
>
> so I need to run the suite on a *86* linux box.

Running GNAT CE 2018 on Debian stretch in a VMWare Fusion VM, I get

		=== acats Summary ===
# of expected passes		2505
# of unexpected failures	15
# of expected failures		1458
# of unresolved testcases	11
# of unsupported tests		116
*** FAILURES: c611a04 c760a02 c760a03 cxd1003 cxd1004 cxd1005 cxd2002
cxd2003 cxd2004 cxd2006 cxd2008 cxd3001 cxd3002 cxd6001 cxd6002

c250002 passes (this is a case-sensitive filesystem), cxag003 has been
fixed.

New fails cxd2002, cxd2003, cxd2004, cxd2008, cxd6001, cxd6002 are
because the target machine was a (VMWare) uniprocessor, so those tests
were allowed to run, and failed presumably because this was again a
non-real-time environment.