Re: Ada Core Technologies and Ada95 Standards

[dewar@cs.nyu.edu (Robert Dewar)]

John McCabe said

A particular employee of a particular Ada compiler vendor (who you
probably know) in a presentation in Waterlooville England on 14th
March 1995 stated that ACVC 2.0 consisted of only the parts of Ada
that were common between Ada 83 and 95. I interpreted this to mean
just the core language but looking back on it I can understand that
this would also mean _parts_ of the specialised needs annexes.

  That particular employee did not know what he or she was talking
  about. You can look for yourself at 2.0, it has MANY tests for
  features in Ada 95 that are not in Ada 83, including all the
  annexes. Probably what either you or the sales person got confused
  over was that for transitional validations, you don't have to pass
  all these "new" tests. So you have to look at the resulting VSR's
  to understand the results.

I was obviously thinking of validation of Ada compilers in the same
way that _my_ software is validated - i.e a full set of test cases
proving that _all_ requirements have been met. If I cannot prove this,
my software is not accepted by my customer.

  100% reliability via testing is only achievable for very simple tasks
  that can be fully specified formally, and for which the number of
  possible independent tests is finite.

  In the case of a compiler, first it it extremely difficult to generate
  the starting point of a formal specification. No formal specificatoin
  exists for Ada, C++ or most other modern complex languages.

  Second, it is trivial to see that no finite set of tests can be complete.
  For example, Ada requires that loops can nest arbitrarily deeply. Suppose
  that the suite has 1000 tests for loop nesting from 1 to a 1000 levels.
  It might still be the case that 1001 loops blows up.

  Or, another example, all possible 64-bit IEEE constants must be
  accurately converted at compile time. It is obviously impossible to test
  this. After all, look at Intel, with all their resources -- they could
  not afford to thoroughly test the divide instruction on the Pentium.
  Just do the calculations, exhaustive testing here is out of the question.

With GNAT you've probably got one of the largest user bases of any
single compiler which can only help. I know GNAT is a very good
"product" (I noticed in a posting some time ago you said GNAT is not a
product but...) but the fact that it is available free of charge would
lead me to be more understanding about its faults. When I pay $40000
for a piece of software development kit, I expect it to work.

  It is a mistake to think that quality of software is proportional to
  price. There is plenty of free software that is good, and plenty
  of expensive software that is terrible, and vice versa.

  Equally, it is a mistake to think that the quality of software
  necessarily depends on the amount of resources invested. It is
  true that the amount of effort invested in GNAT, including the
  effort invested in GCC itself is huge, probably far more than
  for any other compiler, but that in itself is not a guarantee
  of quality, which depends on many factors. 

  In any case, quality speaks for itself, I always advise people
  to judge GNAT on quality not price. When it comes to choosing
  a compiler for a serious project, the only thing that makes
  sense is to choose the best tool for the job.

  P.S. I never said that GNAT was not a product, I said it was not a
  proprietary product! Big difference!

At the end of the day, I want validation to mean that the compiler can
produce working object code from Ada source - and by that I mean the
whole language - a subset is of no use to me. If that is not true of
the compiler then I think that the term used to describe this
examination should not be validation.

  Well that's a matter of terminology. There is no way to be sure that
  any compiler is 100% bug free -- I certainly never met a compiler for
  a complex language that met this criterion. What you can ask for is
  a compiler that is reliable enough that it is not the weak link in
  the chain. NIST incidentally prefers the term certification to describe
  this test-suite oriented testing of compilers.

Robert Dewar
Ada Core Technologies