The Ada Compiler Evaluation System

[brashear@ns1.sw-eng.falls-church.va.us (Philip Brashear)]

There has been a rather acrimonious discussion of compiler validation
and evaluation recently, and I really don't understand the bitterness.
Those of us in the (?quasi?) official Ada compiler testing business
understand "validation" to mean testing against the standard (insofar
as that is possible), simply to get _some_ assurance that the compiler
implements the standard correctly.  We know that we cannot _ensure_
that any compiler does so.  On the other hand, we understand "evaluation"
to mean some attempt to determine whether a compiler meets the performance
and usability needs of a particular project.  (We _assume_ that the
compiler implements the standard correctly.)

Now, I don't claim that these two notions are completely defined, nor that
"our" interpretations are the only reasonable ones.  However, as one who has
led both ACVC (validation suite) and ACES (evaluation suite) development
efforts and the application thereof to various Ada compilers, I can state
that these are the interpretations that have gone into the efforts.  If
there is a need to change the direction of either, then there should be a
discussion along the lines of "I suggest that the validation (or evaluation)
effort be modified as follows:".

Unfortunately, it is very likely that no more DoD funding will be applied
to the evaluation effort, and the funding available for maintenance/upgrade
of the validation suite (and process) will be quite limited.

HOWEVER (he said, finally getting to his point), both the ACVC and the
ACES are extremely useful.  ACVC usage by vendors is pretty much required
(at least for those selling to the U.S. Government); ACES usage isn't.
Some vendors are known to use the ACES; perhaps all do.  There are
organization (such as mine) who are prepared to perform ACES evaluations
and comparisons on a fee basis.  I don't know what one would _expect_ such
a service to cost, so I don't know whether someone would think the price
to be a bargain or a rip-off.  Anyway, I can provide cost estimates to 
organizations having a serious interest.  You may not need the services
of another organization to use the ACES; we think it's now pretty well
documented and fairly user-friendly.  If you're concerned about compiler
selection (actually, implementation selection, including hardware, etc.),
I encourage you to check out the ACES.  It's available to one and all
by anonymous FTP (sw-eng.falls-church.va.us:public/AdaIC/testing/aces/v2.1)
or via the Web (http://sw-eng.falls-church.va.us/AdaIC/testing/aces/).


Phil Brashear
CTA INCORPORATED
5100 Springfield Pike, Suite 100
Dayton, OH  45431
Voice:     (513) 258-0831
Facsimile: (513) 252-3680
brashear@sw-eng.falls-church.va.us
brashear@smtplink.cta.com
brashepw@msrc.wpafb.af.mil
brashear@saber.udayton.edu