Re: limited allocated classwide types

["Randy Brukardt" <randy@rrsoftware.com>]

"Adam Beneschan" <adam@irvine.com> wrote in message 
news:14ff4109-2f02-4dee-9638-68c1befc07c2@hc5g2000vbb.googlegroups.com...
On Nov 4, 2:53 am, Brian Drummond <br...@shapes.demon.co.uk> wrote:
>>
>> But this bug (assuming it is confirmed, as it looks to be) is much closer
>> to the core of the language ... to me it raises the question: has ACATS
>> (test suite) not kept up with the language additions in Ada-2005? And is
>> it being maintained and updated alongside Ada-2012?
>
>I should probably let Randy answer this, but I think the answer to
>your questions is that it hasn't kept up, due to insufficient funding.

Right, both funding and insufficient contributions of tests. (The latter may 
be in part caused by the former.)

>But what I really wanted to point out was that it's not reasonable to
>expect ACATS to catch everything; and the fact that one simple test
>breaks a compiler that passes ACATS isn't itself evidence that there's
>a flaw in ACATS.  I believe ACATS is set up to check each feature and
>rule defined in the language (or something like that), but it can't
>test every combination of every feature.  And this case seems to
>involve a combination of limited controlled types and interfaces (and
>perhaps extended return).  Even if there is, or should be, an ACATS
>test that tests interfaces in conjunction with controlled types, it
>may not occur to the test suite maintainer that there needs to be a
>separate test to do the same thing with limited controlled types,
>because they may not envision that a compiler could work with non-
>limited controlled types but blow up with limited ones.

(Thanks, you saved me a lot of time. :-)

The ACATS is mostly about testing that features are implemented according to 
the Standard. Combinations of features are only tested to the extent that it 
is necessary to verify the implementation. Indeed, there is a huge number of 
possibilities of combinations, and guessing which ones might actually cause 
problems is not likely to have any useful effect on compiler quality.

>Also, I suspect that a major purpose of ACATS is to ensure that the
>compiler vendors understand the language rules and have implemented
>them.  It is not, and cannot, be to ensure that the implementation is
>correct and bug-free in every case.

You don't have to "suspect" that; you can just read the documentation of it. 
Here is the clause "ACATS Purpose" (most of this documentation is little 
changed since the ACVC days for Ada 83):

----

1.1 ACATS Purpose
The purpose of the ACATS is to check whether an Ada compilation system is a 
conforming implementation, i.e., whether it produces an acceptable result 
for every applicable test.

A fundamental goal of conformity assessment (validation) is to promote Ada 
software portability by ensuring consistent processing of Ada language 
features as prescribed by the Ada Standard documents ([Ada95], [TC1], and 
[Amend1]). ACATS tests use language features in contexts and idioms expected 
in production software. While they exercise a wide range of language feature 
uses, they do not and cannot include examples of all possible feature uses 
and interactions.

It is important to recognize that the ACATS tests do not guarantee compiler 
correctness. A compilation system that correctly processes the ACATS tests 
is not thereby deemed error-free, nor is it thereby deemed capable of 
correctly processing all software that is submitted to it.

The ACATS tests do not test the quality of an Ada implementation. In 
particular, ACATS test do not check or report performance parameters (e.g., 
compile-time capacities or run-time speed). They do not check or report for 
characteristics such as the presence and effectiveness of compiler 
optimization. They do not investigate or report compiler or implementation 
choices in cases where the standard allows options.

----

Note that the documentation is included in the ACATS downloads, but isn't 
posted on the internet anywhere that I know of. (Not sure why I didn't put 
it on the ada-auth.org site.)

>> Is there a mechanism for adding ad-hoc test cases like this one to ACATS?
>> One would like to think there is a more systematic effort to add tests
>> alongside features as they are developed.

>I'm wondering if there might be a need for some other public test
>suite that users can contribute to.  I don't think ACATS would be an
>appropriate suite to for just dumping any test cases users find--that
>just doesn't seem like the right place--but perhaps someone could
>start a web site.

I think that would be a great idea. It also would provide a place to mine 
for possible future ACATS tests (it's hard work coming up with test 
scenarios for C-Tests [that is, executable tests]).

                                           Randy.



                             -- Adam