Re: Formal Subprogram Access

["Randy Brukardt" <randy@rrsoftware.com>]

"Simon Wright" <simon@pushface.org> wrote in message 
news:lymv0bk59g.fsf@pushface.org...
> "Randy Brukardt" <randy@rrsoftware.com> writes:
>
>> I have a set of tools for running the grading tool with GNAT. I've
>> been waiting to find some time to finish vetting the results, which
>> I've recently done. [Although I really ought to redo it with GNAT
>> 18.1.] So I'll be posting them soon as "submitted tests" (since
>> compiler-specific stuff doesn't belong in the ACATS proper).
>
> Good news!
>
> For info, my current 8.0.1 20180207 results, vs 4.1f, are
>
> # of expected passes 2500
> # of unexpected failures 9
> # of expected failures 1451
> # of unresolved testcases 11
> # of unsupported tests 121
> *** FAILURES: c250002  cxd1003 cxd1004 cxd1005 cxd4007  cxd2006 cxd3001 
> cxd3002  c611a04

This is similar to the list of C-Tests that the ACATS tools report as 
failed.

> 'unsupported' includes those where there is more than one copy of each
> unit in the input source.
>
> 'unresolved' means visual inspection required, all OK.
>
> c250002 is a Mac file system UTF8 case sensitivity issue, OK on Linux.

I have a similar problem on Windows. I marked this one as a tools bug (since 
I didn't implement the needed UTF-8 support into the ACATS grading tools 
version 1.0), but perhaps it is a real bug with GNAT.

> The cxd failures are I think down to running on a host system, & AdaCore
> would probably mark them as expected failures.

On Windows, I have a somewhat different list of cxd tests that fail, but 
I've blamed that on Windows not having the appropriate behavior. Probably an 
implementer wouldn't test Annex D on such a target (which would potentially 
eliminate the entire set from grading).

> The new one to me is c611a04 (class-wide pre/post conditions); also
> fails with GCC 7.1.0.

I think this test is quite new, and just reflects a bug in GNAT. GNAT, like 
all Ada compilers, fails a decent percentage of new ACATS tests. It's 
unreasonable to expect perfection in this area, and one expects that a 
future version will fix it.

I also have a bunch of failed B-Tests (since the primary point of the 
grading tools was to automate grading of B-Tests); that grading is somewhat 
subjective as any hand grading will be. But there a few old tests that show 
no errors at all; it's hard to imagine how those work unless they work 
better with some other option setting.

There's also a large number of tests that fail grading mainly because the 
tools can't handle where the error messages are placed. Those tests will be 
corrected over time (some in 4.1G, coming soon) but for now one has to just 
put them on the manual grading list that the ACATS tool takes.

One of the big things learned from these automatic grading tools is that it 
is really easy for junk results to creep into typical ACATS grading setups 
(which usually depend on comparing against known-good results). I found 4 
ACATS tests that were marked as passing for Janus/Ada that actually failed. 
Two of those actually reflected compiler bugs introduced in recent years 
(both easy to fix, thank goodness), one was a batch file problem, and one 
probably was just let off of the to-do list (but of course if it isn't on 
the to-do list, it isn't very likely to ever be worked on). Thus I'm not too 
surprised to find similar things for GNAT.

                                         Randy.