Re: Ada Core Technologies and Ada95 Standards

[dewar@cs.nyu.edu (Robert Dewar)]

Ken said

"I know that NPL has a tool that they sell that tests Ada compilers for bugs, tha
t
apparently provides much more coverage than the ACVC. Why should such a tool
exist outside of the validation/certification process?"

This is not true at all, and I guess Ken is only aware of this tool by
rumour, since if he had used it he would now that it is not ni the business
AT ALL of providing coverage testing.

Instead this is a stress testing tool, it generates random very complex
(and generally very unrealistic) examples of expressions and other 
constructs to see if the compiler can be broken by such stress testing.
The tool is incidentally available from NPL to users, so Ken it is 
certainly something you could use to test a compiler yourself.

In fact, I talked to Brian Wichman (the author of this tool and a similar
one for Pascal), and the results they have obtained with these tools are
quite surprising at least to me, in the extent to which they show quality
differences between Ada and Pascal compilers. Most (all?) of the Pascal
compilers they have tested have exhibited safety defects (defined as the
generation of incorrect code). None of the Ada compilers have shown
safety defects -- they have managed to break them but not persauded
then to generate wrong code.

Now in practice, I would expect that big projects such as Ken's can point
to safety defects (defined this way) in the compilers they have used, and
just as the ACVC cannot 100% guarantee conformance, the NPL tests cannot
100% guarantee safety, but they are a measure. I find it interesting that
the Ada compilers fair so much better than the Pascal compilers. Brian at
least ascribes this at least in part to the ACVC process.