Re: Five Years After

["Randy Brukardt" <randy@rrsoftware.com>]

"Maciej Sobczak" <see.my.homepage@gmail.com> wrote in message 
news:e7a16703-b24e-42aa-81fd-7026932d6c32@googlegroups.com...
>> Everyone agrees that the ACATS test suite offers a lot of value to both
>> implementers and users (implementers because an independently sourced and
>> trustworthy set of tests is very helpful to creating and maintaining an 
>> Ada
>> compiler -- it is especially good IMHO in preventing errors of omission,
>> which are often the hardest to find;
>
>Sorry, but I cannot agree here. There is nothing that saves ACATS authors 
>from errors of omission.

Of course there is -- extensive coverage documentation. Of course that's not 
going to find *every* possible omission, but it tries to check that every 
sentence in the Standard is tested at least once.

And please note that "valuable" is not the same as "the only thing anyone 
ever needs"!! The ACATS is not now, and never was, intended to find compiler 
bugs per se. It is intended to prove a level of compliance with the Ada 
Standard - for instance that a task terminates when it is supposed to, and 
not sooner or later. It can't try to check every combination of features - 
such that a task given in a block in a generic subprogram that has as 
discriminant that is a generic formal parameter terminates on time -- that 
would be an impossible task.

> The language is itself so complex that creating a complete (and I really 
> mean
> formally complete) set of requirements is impractical and without such a 
> set it
> is not possible to create a complete (again, I mean it) set of tests.

Nobody is trying to do this, because it would be impossible (practically) 
even if one had unlimited money.

Real compiler testing has to be a combination of black box tests (the ACATS 
test suite being a great example) and tests designed to ensure code coverage 
and other sorts of white box tests. And I think every vendor also has a 
bunch tests created from user bug reports, mainly to try to ensure 
regressions don't happen. But there is no single technique that could 
provide anything like perfection for any real programming language. (Maybe 
for toy subsets.)

> As a result, there are areas of the language that were quit well 
> understood
> by language designers and more or less well understood by language users,
> that are not covered by ACATS tests at all and therefore not properly
> implemented by compilers. I have myself found several compiler bugs by 
> only
> playing with and shuffling ideas from textbook examples (and we have
> discussed them here on this group).

Repeat: The ACATS is not intended to find compiler bugs, only gross mistakes 
in the implementation of Ada features. (The obvious example is not 
implementing something at all.)

Besides, the ACATS only covers about 45% of the first level objectives, 
based on the parts of the Standard that I've analyzed to date. (Coverage 
might be a bit better overall, as some of the very basic sections haven't 
been analyzed yet, like most of the statements in Chapter 5 and expressions 
in Chapter 4.) (A lot of those aren't very interesting or likely, but 
untested is always a possibility for an omission or gross error.) There 
isn't ever going to be enough funding to close that gap.

>> This certainly is true since I'm paid money every year to maintain and
>> enhance the test suite
>
> Ironically, the bare fact that you are still being paid for this work is 
> itself a proof
> that these tests are still not complete. ;-) ;-) ;-)

Completeness is a goal that could never really be reached (probably some 
part of the objectives would be untestable in practice). And it doesn't 
matter, the test suite is still *valuable* even if not *perfect*.

>But these facts do not always translate into satisfactory user experience.

The ACATS has absolutely nothing to do with the user experience.

...
> In which case a different approach to compiler validation might be needed.

It will never be the least bit hard to find a bug in a compiler (for any 
realistic language). Surely not in an Ada compiler, C++ compiler, Java 
compiler, and so on.

At least not until humans are not allowed near computers. ;-)

                                               Randy.