Re: Five Years After

[Maciej Sobczak <see.my.homepage@gmail.com>]

> Everyone agrees that the ACATS test suite offers a lot of value to both 
> implementers and users (implementers because an independently sourced and 
> trustworthy set of tests is very helpful to creating and maintaining an Ada 
> compiler -- it is especially good IMHO in preventing errors of omission, 
> which are often the hardest to find;

Sorry, but I cannot agree here. There is nothing that saves ACATS authors from errors of omission. The language is itself so complex that creating a complete (and I really mean formally complete) set of requirements is impractical and without such a set it is not possible to create a complete (again, I mean it) set of tests. As a result, there are areas of the language that were quit well understood by language designers and more or less well understood by language users, that are not covered by ACATS tests at all and therefore not properly implemented by compilers. I have myself found several compiler bugs by only playing with and shuffling ideas from textbook examples (and we have discussed them here on this group).

> This certainly is 
> true since I'm paid money every year to maintain and enhance the test 
> suite

Ironically, the bare fact that you are still being paid for this work is itself a proof that these tests are still not complete. ;-) ;-) ;-)

> Ada compilers accept essentially the same language 
> since they've all been tested with the same set of tests).

Unfortunately, no. Every single version of whatever compiler accepts a different language than its previous version. This is how the bugs are fixed (or introduced), among others.

The problem is - non of our programs is really a mashup of unit tests from ACATS, the state space is too big for this. It is really good that the tests run fine. Heck - it's really good that they exist at all! But these facts do not always translate into satisfactory user experience. Ultimately, my intent is to compile my own programs, not the tests - and apparently, the whole concept of compiler testing is too big to extrapolate from a set of unit tests to a set of all possible user programs. Maybe Ada as a language has already crossed the border at which such an approach is valid at all.
In which case a different approach to compiler validation might be needed.

-- 
Maciej Sobczak * http://www.inspirel.com