Re: Ada Core Technologies and Ada95 Standards

[john@assen.demon.co.uk (John McCabe)]

dewar@cs.nyu.edu (Robert Dewar) wrote:
<..snip...>

>Wrong! The validation suite does contain tests for all parts of Ada 95
>including all the special needs annexes, and this is true "at the moment"
>(where DO these rumours come from? :-)

A particular employee of a particular Ada compiler vendor (who you
probably know) in a presentation in Waterlooville England on 14th
March 1995 stated that ACVC 2.0 consisted of only the parts of Ada
that were common between Ada 83 and 95. I interpreted this to mean
just the core language but looking back on it I can understand that
this would also mean _parts_ of the specialised needs annexes.

>It is certainly true that the
>initial release of ACVC 2.0 and now 2.0.1 does not thoroughly cover
>all new parts of the language, but as any Ada 95 compiler implementor
>can tell you, they are definitely non-trivial, and any compiler passing
>all or nearly all of these tests is a pretty complete Ada 95 compiler.

Mmmm. Seems a bit contradictory ("all parts of Ada95" and "not all new
parts")

>As for Ada 83 validation not proving much, if you feel this way, probably
<..snip..>

I was obviously thinking of validation of Ada compilers in the same
way that _my_ software is validated - i.e a full set of test cases
proving that _all_ requirements have been met. If I cannot prove this,
my software is not accepted by my customer.

In the tools and utilities market this level of proof does not seem to
be required. I design and implement systems for satellite instrument
control. The software will have 1 user. I cannot just put a 1st
version of the software onto a satellite and wait for the user to send
me bug reports because by that time the chances are that the whole
system could have failed and $250M worth of satellite is lying at the
bottom of the Pacific Ocean.

>What does validation do? It makes sure that the vendor has implemented
>the entire language without significant gaps, and that the vendor has
>implemented large parts of the language (those parts tested) accurately.
>As a result, it is a good guarantee that the vendor undrstands the
>language completely and thoroughly.

There is a large difference between implementing the entire language
and implementing it accurately. I find the number of faults with basic
language handling in my present compiler rather disturbing.

>Can a test suite do more than this? No! Can it do a better or worse job
>of this? Sure. We think the ACVC 2.1 suite will turn out to be more
>effective, because we have learned something in 12 years! In particular
>we (the ACVC team and the reviewers) believe that the orientation to
>more user-oriented testing will be helpful in this regard (compare some
>typical 2.0 test with 1.11 tests, and you will see that the 2.0 tests
>are much more like real programs -- the test writer testing a particular
>feature thinks "how would this feature be used in a real program", and
>constructs a real program to answer that question.

That's good and basically the way it should be.

>HOWEVER, although the suite will, we believe, be even more effective
>than the 1.11 suite, no one would claim that it guarantees 100%
>conformance or usability. If you hear anyone saying this, beware!
>they do not know what they are talking about.

I can accept that it is very difficult to prove a tool such as this
completely but when I buy a _validated_ Ada compiler it is because I
want to compile _valid_ _Ada_ code, not a subset of it!

>There are many ways to evaluate a compiler. GNAT is validated, but it has
<..snip..>

I agree entirely with what you say here. It is obvious that the more
users of a compiler, the more likely the bugs are found and sorted
early on. That has been a problem with our compiler (MIL-STD-1750A
version) because the user base is tiny. What is more disturbing
however is that for every bug that seems to get fixed, the new release
seems to contain even more!

With GNAT you've probably got one of the largest user bases of any
single compiler which can only help. I know GNAT is a very good
"product" (I noticed in a posting some time ago you said GNAT is not a
product but...) but the fact that it is available free of charge would
lead me to be more understanding about its faults. When I pay $40000
for a piece of software development kit, I expect it to work.

At the end of the day, I want validation to mean that the compiler can
produce working object code from Ada source - and by that I mean the
whole language - a subset is of no use to me. If that is not true of
the compiler then I think that the term used to describe this
examination should not be validation.



Best Regards
John McCabe <john@assen.demon.co.uk>