From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Thread: 103376,43127f177a55dc41 X-Google-NewGroupId: yes X-Google-Attributes: gida07f3367d7,domainid0,public,usenet X-Google-Language: ENGLISH,ASCII-7-bit Received: by 10.68.25.194 with SMTP id e2mr2080615pbg.7.1320726341994; Mon, 07 Nov 2011 20:25:41 -0800 (PST) MIME-Version: 1.0 Path: h5ni11166pba.0!nntp.google.com!news2.google.com!volia.net!news2.volia.net!feed-A.news.volia.net!news.ecp.fr!news.jacob-sparre.dk!pnx.dk!jacob-sparre.dk!ada-dk.org!.POSTED!not-for-mail From: "Randy Brukardt" Newsgroups: comp.lang.ada Subject: Re: limited allocated classwide types Date: Mon, 7 Nov 2011 22:25:38 -0600 Organization: Jacob Sparre Andersen Research & Innovation Message-ID: References: <0ed43f83-40e7-46d3-8cc4-e1c41f500d28@c1g2000vbw.googlegroups.com> <14ff4109-2f02-4dee-9638-68c1befc07c2@hc5g2000vbb.googlegroups.com> NNTP-Posting-Host: static-69-95-181-76.mad.choiceone.net X-Trace: munin.nbi.dk 1320726340 21691 69.95.181.76 (8 Nov 2011 04:25:40 GMT) X-Complaints-To: news@jacob-sparre.dk NNTP-Posting-Date: Tue, 8 Nov 2011 04:25:40 +0000 (UTC) X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.5931 X-RFC2646: Format=Flowed; Original X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Xref: news2.google.com comp.lang.ada:14338 Date: 2011-11-07T22:25:38-06:00 List-Id: "Adam Beneschan" wrote in message news:14ff4109-2f02-4dee-9638-68c1befc07c2@hc5g2000vbb.googlegroups.com... On Nov 4, 2:53 am, Brian Drummond wrote: >> >> But this bug (assuming it is confirmed, as it looks to be) is much closer >> to the core of the language ... to me it raises the question: has ACATS >> (test suite) not kept up with the language additions in Ada-2005? And is >> it being maintained and updated alongside Ada-2012? > >I should probably let Randy answer this, but I think the answer to >your questions is that it hasn't kept up, due to insufficient funding. Right, both funding and insufficient contributions of tests. (The latter may be in part caused by the former.) >But what I really wanted to point out was that it's not reasonable to >expect ACATS to catch everything; and the fact that one simple test >breaks a compiler that passes ACATS isn't itself evidence that there's >a flaw in ACATS. I believe ACATS is set up to check each feature and >rule defined in the language (or something like that), but it can't >test every combination of every feature. And this case seems to >involve a combination of limited controlled types and interfaces (and >perhaps extended return). Even if there is, or should be, an ACATS >test that tests interfaces in conjunction with controlled types, it >may not occur to the test suite maintainer that there needs to be a >separate test to do the same thing with limited controlled types, >because they may not envision that a compiler could work with non- >limited controlled types but blow up with limited ones. (Thanks, you saved me a lot of time. :-) The ACATS is mostly about testing that features are implemented according to the Standard. Combinations of features are only tested to the extent that it is necessary to verify the implementation. Indeed, there is a huge number of possibilities of combinations, and guessing which ones might actually cause problems is not likely to have any useful effect on compiler quality. >Also, I suspect that a major purpose of ACATS is to ensure that the >compiler vendors understand the language rules and have implemented >them. It is not, and cannot, be to ensure that the implementation is >correct and bug-free in every case. You don't have to "suspect" that; you can just read the documentation of it. Here is the clause "ACATS Purpose" (most of this documentation is little changed since the ACVC days for Ada 83): ---- 1.1 ACATS Purpose The purpose of the ACATS is to check whether an Ada compilation system is a conforming implementation, i.e., whether it produces an acceptable result for every applicable test. A fundamental goal of conformity assessment (validation) is to promote Ada software portability by ensuring consistent processing of Ada language features as prescribed by the Ada Standard documents ([Ada95], [TC1], and [Amend1]). ACATS tests use language features in contexts and idioms expected in production software. While they exercise a wide range of language feature uses, they do not and cannot include examples of all possible feature uses and interactions. It is important to recognize that the ACATS tests do not guarantee compiler correctness. A compilation system that correctly processes the ACATS tests is not thereby deemed error-free, nor is it thereby deemed capable of correctly processing all software that is submitted to it. The ACATS tests do not test the quality of an Ada implementation. In particular, ACATS test do not check or report performance parameters (e.g., compile-time capacities or run-time speed). They do not check or report for characteristics such as the presence and effectiveness of compiler optimization. They do not investigate or report compiler or implementation choices in cases where the standard allows options. ---- Note that the documentation is included in the ACATS downloads, but isn't posted on the internet anywhere that I know of. (Not sure why I didn't put it on the ada-auth.org site.) >> Is there a mechanism for adding ad-hoc test cases like this one to ACATS? >> One would like to think there is a more systematic effort to add tests >> alongside features as they are developed. >I'm wondering if there might be a need for some other public test >suite that users can contribute to. I don't think ACATS would be an >appropriate suite to for just dumping any test cases users find--that >just doesn't seem like the right place--but perhaps someone could >start a web site. I think that would be a great idea. It also would provide a place to mine for possible future ACATS tests (it's hard work coming up with test scenarios for C-Tests [that is, executable tests]). Randy. -- Adam