From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,INVALID_MSGID autolearn=no autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,e4e62e0a73fb6667 X-Google-Attributes: gid103376,public From: dewar@cs.nyu.edu (Robert Dewar) Subject: Re: The Ada Compiler Evaluation System Date: 1996/04/27 Message-ID: #1/1 X-Deja-AN: 151684498 references: <4l2nt1$p4k@ns1.sw-eng.falls-church.va.us> <31761BD5.7D11@lmtas.lmco.com> <317B7757.4849@lmtas.lmco.com> <3180CC5C.321F@lmtas.lmco.com> organization: Courant Institute of Mathematical Sciences newsgroups: comp.lang.ada Date: 1996-04-27T00:00:00+00:00 List-Id: Ken said "Been there, done that. Even after I quoted excerpts from the ACVC 2.0 documentation, the response has been to RTFM and to "stop bashing those poor ACVC folks."" Ken, we are not talking about the "ACVC 2.0 documentation" we are suggesting that you look at the tests themselves. If you have "been there, done that", then I would really be interested in your opinion of the new testing approach (and so would the ACVC test development team!) P.S. on another matter, I think the important distinction that Gary was trying to make was that you can pass or fail the ACVC tests, they are obvjective in that sense. You cannot pass or fail the ACEC tests -- any more than you can pass or fail the SpecMark (which incidentally is a TERRIBLY misleading indication of microprocessor performance, as everyone in the industry knows, but everyone keeps advertising it, because users do indeed depend on it -- this is one of the well known problems with standard performance tests). You also said that you thought all compilers should be run through the ACEC tests. That's certainly reasonable if the market requires it. It might even be reasonable for the DoD to fund this. It certainly *is* reasonable, and very common, for customers to require ACEC testing, and they don't "each pay for the" tests, the vendor runs them once (why would you think things work differently). Of course some vendors might not be inerested in the ACEC tests. We never bothered for instance to run Ada/Ed through the ACEC tests, except as basic functionality tests, since the performance of Ada/Ed was not particularly relevant to its users, or, more accurately, maybe it was relevant, but performance was *not* a salient feature of Ada/Ed. Other vendors may simply not be interested in customers who require ACES testing, and that's quite a reasonable position (they won't be able to sell to these customers). You can't even force all vendors to validate, let alone run other tests in any compleely general way. You *can* and *should* request vendros to meet your needs, and requesting ACEC test results is certainly reasonable (it is a bit of a surprise to me that the F22 program did not require ACEC testing). As for publishing test results, as far as I know ACEC test results can be published -- this is not true of all test suites, there are often restrictions. Spec results can only be published in a certain manner, and as far as I know results from the NPL stress tests cannot be published at all (although this might have changed) -- basically the idea behind such restrictions is to avoid over-focus on the results of a particular suite, or to avoid its use in advertising. That decision is of course up to the vendor of the test suite.