From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable autolearn_force=no version=3.4.4 Path: eternal-september.org!reader01.eternal-september.org!reader02.eternal-september.org!feeder.eternal-september.org!nntp-feed.chiark.greenend.org.uk!ewrotcd!newsfeed.xs3.de!io.xs3.de!news.jacob-sparre.dk!franka.jacob-sparre.dk!pnx.dk!.POSTED.rrsoftware.com!not-for-mail From: "Randy Brukardt" Newsgroups: comp.lang.ada Subject: Re: Formal Subprogram Access Date: Wed, 14 Feb 2018 22:56:21 -0600 Organization: JSA Research & Innovation Message-ID: References: Injection-Date: Thu, 15 Feb 2018 04:56:22 -0000 (UTC) Injection-Info: franka.jacob-sparre.dk; posting-host="rrsoftware.com:24.196.82.226"; logging-data="13275"; mail-complaints-to="news@jacob-sparre.dk" X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.5931 X-RFC2646: Format=Flowed; Original X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.7246 Xref: reader02.eternal-september.org comp.lang.ada:50449 Date: 2018-02-14T22:56:21-06:00 List-Id: "Simon Wright" wrote in message news:lymv0bk59g.fsf@pushface.org... > "Randy Brukardt" writes: > >> I have a set of tools for running the grading tool with GNAT. I've >> been waiting to find some time to finish vetting the results, which >> I've recently done. [Although I really ought to redo it with GNAT >> 18.1.] So I'll be posting them soon as "submitted tests" (since >> compiler-specific stuff doesn't belong in the ACATS proper). > > Good news! > > For info, my current 8.0.1 20180207 results, vs 4.1f, are > > # of expected passes 2500 > # of unexpected failures 9 > # of expected failures 1451 > # of unresolved testcases 11 > # of unsupported tests 121 > *** FAILURES: c250002 cxd1003 cxd1004 cxd1005 cxd4007 cxd2006 cxd3001 > cxd3002 c611a04 This is similar to the list of C-Tests that the ACATS tools report as failed. > 'unsupported' includes those where there is more than one copy of each > unit in the input source. > > 'unresolved' means visual inspection required, all OK. > > c250002 is a Mac file system UTF8 case sensitivity issue, OK on Linux. I have a similar problem on Windows. I marked this one as a tools bug (since I didn't implement the needed UTF-8 support into the ACATS grading tools version 1.0), but perhaps it is a real bug with GNAT. > The cxd failures are I think down to running on a host system, & AdaCore > would probably mark them as expected failures. On Windows, I have a somewhat different list of cxd tests that fail, but I've blamed that on Windows not having the appropriate behavior. Probably an implementer wouldn't test Annex D on such a target (which would potentially eliminate the entire set from grading). > The new one to me is c611a04 (class-wide pre/post conditions); also > fails with GCC 7.1.0. I think this test is quite new, and just reflects a bug in GNAT. GNAT, like all Ada compilers, fails a decent percentage of new ACATS tests. It's unreasonable to expect perfection in this area, and one expects that a future version will fix it. I also have a bunch of failed B-Tests (since the primary point of the grading tools was to automate grading of B-Tests); that grading is somewhat subjective as any hand grading will be. But there a few old tests that show no errors at all; it's hard to imagine how those work unless they work better with some other option setting. There's also a large number of tests that fail grading mainly because the tools can't handle where the error messages are placed. Those tests will be corrected over time (some in 4.1G, coming soon) but for now one has to just put them on the manual grading list that the ACATS tool takes. One of the big things learned from these automatic grading tools is that it is really easy for junk results to creep into typical ACATS grading setups (which usually depend on comparing against known-good results). I found 4 ACATS tests that were marked as passing for Janus/Ada that actually failed. Two of those actually reflected compiler bugs introduced in recent years (both easy to fix, thank goodness), one was a batch file problem, and one probably was just let off of the to-do list (but of course if it isn't on the to-do list, it isn't very likely to ever be worked on). Thus I'm not too surprised to find similar things for GNAT. Randy.