Re: The Ada Compiler Evaluation System

[dewar@cs.nyu.edu (Robert Dewar)]

Ken said

"Been there, done that. Even after I quoted excerpts from the ACVC 2.0
documentation, the response has been to RTFM and to "stop bashing those
poor ACVC folks.""

Ken, we are not talking about the "ACVC 2.0 documentation" we are 
suggesting that you look at the tests themselves. If you have "been there,
done that", then I would really be interested in your opinion of the
new testing approach (and so would the ACVC test development team!)

P.S. on another matter, I think the important distinction that Gary was
trying to make was that you can pass or fail the ACVC tests, they are
obvjective in that sense. You cannot pass or fail the ACEC tests -- any
more than you can pass or fail the SpecMark (which incidentally is
a TERRIBLY misleading indication of microprocessor performance, as
everyone in the industry knows, but everyone keeps advertising it,
because users do indeed depend on it -- this is one of the well known
problems with standard performance tests).

You also said that you thought all compilers should be run through the
ACEC tests. That's certainly reasonable if the market requires it. It
might even be reasonable for the DoD to fund this. It certainly *is*
reasonable, and very common, for customers to require ACEC testing,
and they don't "each pay for the" tests, the vendor runs them once
(why would you think things work differently).

Of course some vendors might not be inerested in the ACEC tests. We never
bothered for instance to run Ada/Ed through the ACEC tests, except as
basic functionality tests, since the performance of Ada/Ed was not
particularly relevant to its users, or, more accurately, maybe it
was relevant, but performance was *not* a salient feature of Ada/Ed.
Other vendors may simply not be interested in customers who require
ACES testing, and that's quite a reasonable position (they won't
be able to sell to these customers). 

You can't even force all vendors to validate, let alone run other tests
in any compleely general way. You *can* and *should* request vendros
to meet your needs, and requesting ACEC test results is certainly
reasonable (it is a bit of a surprise to me that the F22 program did
not require ACEC testing).

As for publishing test results, as far as I know ACEC test results can
be published -- this is not true of all test suites, there are often
restrictions. Spec results can only be published in a certain manner,
and as far as I know results from the NPL stress tests cannot be
published at all (although this might have changed) -- basically
the idea behind such restrictions is to avoid over-focus on the
results of a particular suite, or to avoid its use in advertising.
That decision is of course up to the vendor of the test suite.