Re: Ada Core Technologies and Ada95 Standards

[dewar@cs.nyu.edu (Robert Dewar)]

John McCabe says

"This paragraph confuses me.

I believe that if the compiler vendors were producing high quality
products that conformed to the language, then passing the ACVC suite
should be a cinch.

What you seem to be saying is that compiler vendors are assigning
resources to ensure that they pass the ACVC suite, and in doing so are
compromising the quality of their product."

This is a very common misconception among people who don't know language
design or compiler implementation very well. Indeed it is the same
misconception that has lead people to misinterprete what conformance 
testing is all about.

Perhaps I can put it this way. Suppose a vendor has resources to do exacty
one of the following two tasks:

1. Rewrite the loop optimizer so that all loops run faster

2. Rewrite the handling of static expressions so that one very obscure
test in the ACVC suite which has never shown up in a custoer programer
and is very unlikely *ever* to show up in customer programs, under the
condition that this rewriting is extensive and will likely cause
regressions (in programs other than ACVC tests).

Which do YOU think would contribute more to quality for most users?

That's the potential trouble with mandated testing, it elevates very
unimportant problems to maximum priority without any regard to the
importance or impact of these problems. 

If we follow Ken's repeated request, and extend the scope of mandatory
testing, then we distort things still more. That's the risk.

The DoD policy in this area is that requiring the ACVC conformance testing
is as far as it is desirable to go for general requirements. This policy
of course recognizes that for certain purposes, other kinds of testing
may be useful, which is why DoD has suppoted the development of the ACES
suite. It is up to a project officer to specify this as a requirement
if this is needed (such requirements are not uncommon). Ken, if your
procurement did not specify this requirement, all one can ask is why
not? Do you really need the DoD to tell you what testing needs to be done?
In the commercial marketplace, the market determines what testing is
desirable (for instance a lot of the C++ commercial market is comfortable
with no testing whatsoever), but in other contexts he commercial marketplace
requires testing, e.g. many commercial COBOL customers wlil only used
NIST certified compilers. Why is it that DoD customers can't work the
same way.

By the way Ken, you ask how the DoD has determined that it is reasonable
to generally require the ACVC testing and nothing more? I find it a bit
odd that a DoD contractor should be asking this question to someone
outside -- why not ask within the DoD, it's their policy!

P.S. when I used critical in talking about non-critical banking applications,
I was abbreviating not for mission-critical, but for safety-critical. Sorry
for not being clear. But to clarify my point here. A banking applicatoin
may well not care about ACES testing because they don't care about
performance, and their own domain specific testing (e.g. actual testing
of the application in question) shows that a given compiler works
adeqately for their purposes.