Re: Ada Core Technologies and Ada95 Standards

[dewar@cs.nyu.edu (Robert Dewar)]

iKen Garlington says

"
It's probably just ignorance on my part about the ACVC process, but I don't
get that same sense of rigor in the ACVC design. A lot of what's known about
good processes for software testing (as documented by Beizer and others)
isn't apparent from what little I've heard about the ACVC, and from reading the
old ACVC 1.0 tests."

Yup, you are right on target (it is ignorance on your part!).

"1. We have a requirements specification that uniquely identifies each
requirement.
"

Yes, of course this was what was done for ACVC version 1 (ever read
the implementors guide? I guess not! This was the requirements spec
for the testing)

"2. We have a test or set of tests which can be traced back to each requirement."

Yes, of course this was done (don't you see the objectives in the test
traced back to the requirements, you said you read the tests).

"3. We have consultations with the end user of the system to see if the tests
are adequate, and reflect the usage of the system.

"

This is *especially* being done for the new ACVC 2 (I guess you are
unfamiliar with the process here).

Your comments on white box testing are not relevant for a general
validatoin facility, though of course for a given compiler, these
kind of procedures are followed.

It would not be practical to incorporate all test progrms for all bugs
found in all compilers into the ACVC (it would rapidly have tens of
thousands of tests, and become completely unmanagable). For example,
the GNAT regression tests now are larger than the whole ACVC test
suite by a considerable factor. Also, the effort of taking every
bug and putting it into ACVC form is out of the question.

The Ada ACVC suite is by fac the most comprehensive test suite ever
generated for a programming language. THe fact that it is stlil not
truly comprehensive just serves to emphasize how complex compilers
for modern large languages are.

Ken, a minimal effort on your part invested in learning about the ACVC
process would seem a worthwhile effort, and would certainly make your
comments about the ACVC more informed. Have you even read John
Goodenough's papers on the subject?