Re: limited allocated classwide types

[Brian Drummond <brian@shapes.demon.co.uk>]

On Mon, 07 Nov 2011 22:25:38 -0600, Randy Brukardt wrote:

> "Adam Beneschan" <adam@irvine.com> wrote in message
> 
news:14ff4109-2f02-4dee-9638-68c1befc07c2@hc5g2000vbb.googlegroups.com...
> On Nov 4, 2:53 am, Brian Drummond <br...@shapes.demon.co.uk> wrote:

>>> has ACATS (test suite) not kept up with the language additions in
>>> Ada-2005? And is it being maintained and updated alongside Ada-2012?
>>
>>I should probably let Randy answer this, but I think the answer to your
>>questions is that it hasn't kept up, due to insufficient funding.
> 
> Right, both funding and insufficient contributions of tests. (The latter
> may be in part caused by the former.)

I can understand that! And I appreciate that much high quality work goes 
into the Standard; perhaps expecting ACATS to keep up is being greedy!

>> and the fact that one simple test
>>breaks a compiler that passes ACATS isn't itself evidence that there's a
>>flaw in ACATS.  I believe ACATS is set up to check each feature and rule
>>defined in the language (or something like that), but it can't test
>>every combination of every feature.  And this case seems to involve a
>>combination of limited controlled types and interfaces (and perhaps
>>extended return).

I agree that it's impractical to test every combination of features. 

But perhaps it is possible to test, in addition to every feature 
individually, the sum of all of them. 

(Of course I admit even then, it may not have caught this particular 
corner case)

> It is important to recognize that the ACATS tests do not guarantee
> compiler correctness. A compilation system that correctly processes the
> ACATS tests is not thereby deemed error-free, nor is it thereby deemed
> capable of correctly processing all software that is submitted to it.

Which does not negate the value of ACATS in any way.

>>> Is there a mechanism for adding ad-hoc test cases like this one to
>>> ACATS? One would like to think there is a more systematic effort to
>>> add tests alongside features as they are developed.

I should have been clearer that there were two independent questions 
here ...

(1) can I or Simon B or Joe Bloggs submit tests <somewhere>?

This has been pretty well answered : Simon W pointed to submitted not-yet-
ACATS tests. And...
>>I'm wondering if there might be a need for some other public test suite
>>that users can contribute to.  I don't think ACATS would be an
>>appropriate suite to for just dumping any test cases users find--that
>>just doesn't seem like the right place--but perhaps someone could start
>>a web site.
> 
> I think that would be a great idea. It also would provide a place to
> mine for possible future ACATS tests (it's hard work coming up with test
> scenarios for C-Tests [that is, executable tests]).

I absolutely agree that ACATS shouldn't be a dumping ground for random 
testcases ... but, if there were such an extensible test suite, it would 
inevitably contain the occasional nugget worth polishing up and adding to 
the canon.

Perhaps a starting point is to GPL any testcases we submit against (FSF 
or GPL) Gnat, so that they can become part of the public record. (I 
understand that commercial bug reports against commercial compilers will 
be treated differently!) 

The second question : 
(2) when adding a new feature or language revision, is there a systematic 
effort to add tests for it? 
I think has been answered by the remarks on funding...

Thanks,
- Brian