Re: Ada Core Technologies and Ada95 Standards

[Ken Garlington <garlingtonke@lmtas.lmco.com>]

Robert Dewar wrote:
> 
> Why not study the tests?
> Why not study the process being used to construct ACVC 2.1?

[to determine if the new tests reflect "common" usage]

Well, I did. Unfortunately, I don't know what represents common usage.
I only know how I (and my group use Ada). Is it common usage? Beats me.
This was exactly my comment: Without some sort of survey, etc. how can
_any_ one person or small group of people know what represents common usage?

There's another issue. Common usage today is based on Ada 83 (unless you
wish to claim that most existing Ada code was developed using Ada 95).
Therefore, even if such a survey was done, how could it predict how people
will be using Ada 95 unique features?

> The resulting test is then
> reviewed by the ACVC review team, which represents implementors and
> users, to see how well it seems to match potential use.

How does this team represent me, if they have never contacted me? Or have
they contacted most users, and I represent some irrelevant minority?

> It is hard to establish objective criterion for how well one is doing
> in this process. The review group certainly has not found anyway of
> pre-establishing what will turn out to be typical usage.

Exactly. EXACTLY. I don't how they _could_ do this.

Granted, it's a good thing that ACVC is at least asking the question,
"Does this represent real use?" My comment, as you have finally acknowledged,
is that it's not clear how they develop a meaningful answer.

> What *is* encouraging is the following, which actually could possibly
> be quantied from our data, with a lot of work.
> 
> When we make proposed changes to GNAT, we run three kinds of tests:
> 
>  Our main test suite, which is surely user-oriented, since it is mostly
>  user code.
> 
>  the old ACVC tests
> 
>  the new ACVC tests
> 
> in pratice we find the general rests on the new ACVC tests closely mirror
> the general results on the main test suite.

That _is_ encouraging. In fact, it's so encouraging that I think we should
do this for all vendor regression suites, to further build confidence in the
ACVC. In fact, let's be bold. If there is a vendor regression suite that _fails_
to generally track the ACVC results, then there should be some requirement to 
propose an update to the ACVC based on that result. If the divergence were due
to some inapplicable condition, then of course the ACVC should not be updated.
However, for the other cases, it should.

Good idea! Wish I'd thought of it.

> On the other hand, the old ACVC tests seem to pick up a somewhat separate
> set of errors, and often yield conflicting results (lots of problems when
> the other two sets show none, or no problems when the other two sets
> show lots).

Of course, there is also the possibility that those results, while not applicable
to GNAT users, might be very relevant to users from other vendors. How do we know
whether those old tests still have value?

-- 
LMTAS - "Our Brand Means Quality"