Re: Ada Core Technologies and Ada95 Standards

[dewar@cs.nyu.edu (Robert Dewar)]

Ken asked:

"This is certainly true, and one of the questions I asked. If the ACVC is
changing to be more user-oriented, how does the ACVC writer have this
understanding of how the compiler will be used? If his understanding
is in error, then the tests will still be focusing on the wrong things."

Why not study the tests?
Why not study the process being used to construct ACVC 2.1?

This development is not being done in a vacuum. 

The basic answer is that each feature, instead of being tested in a
formalistic way is instead resulting in the question: "How would this
feature be used in a real program?", a question that was never even
asked in the context of ACVC version 1. The resulting test is then
reviewed by the ACVC review team, which represents implementors and
users, to see how well it seems to match potential use. 

It is hard to establish objective criterion for how well one is doing
in this process. The review group certainly has not found anyway of
pre-establishing what will turn out to be typical usage.

What *is* encouraging is the following, which actually could possibly
be quantied from our data, with a lot of work.

When we make proposed changes to GNAT, we run three kinds of tests:

 Our main test suite, which is surely user-oriented, since it is mostly
 user code.

 the old ACVC tests

 the new ACVC tests

in pratice we find the general rests on the new ACVC tests closely mirror
the general results on the main test suite.

On the other hand, the old ACVC tests seem to pick up a somewhat separate
set of errors, and often yield conflicting results (lots of problems when
the other two sets show none, or no problems when the other two sets 
show lots).