The Ada Compiler Evaluation System

The Ada Compiler Evaluation System

[Philip Brashear]

There has been a rather acrimonious discussion of compiler validation
and evaluation recently, and I really don't understand the bitterness.
Those of us in the (?quasi?) official Ada compiler testing business
understand "validation" to mean testing against the standard (insofar
as that is possible), simply to get _some_ assurance that the compiler
implements the standard correctly.  We know that we cannot _ensure_
that any compiler does so.  On the other hand, we understand "evaluation"
to mean some attempt to determine whether a compiler meets the performance
and usability needs of a particular project.  (We _assume_ that the
compiler implements the standard correctly.)

Now, I don't claim that these two notions are completely defined, nor that
"our" interpretations are the only reasonable ones.  However, as one who has
led both ACVC (validation suite) and ACES (evaluation suite) development
efforts and the application thereof to various Ada compilers, I can state
that these are the interpretations that have gone into the efforts.  If
there is a need to change the direction of either, then there should be a
discussion along the lines of "I suggest that the validation (or evaluation)
effort be modified as follows:".

Unfortunately, it is very likely that no more DoD funding will be applied
to the evaluation effort, and the funding available for maintenance/upgrade
of the validation suite (and process) will be quite limited.

HOWEVER (he said, finally getting to his point), both the ACVC and the
ACES are extremely useful.  ACVC usage by vendors is pretty much required
(at least for those selling to the U.S. Government); ACES usage isn't.
Some vendors are known to use the ACES; perhaps all do.  There are
organization (such as mine) who are prepared to perform ACES evaluations
and comparisons on a fee basis.  I don't know what one would _expect_ such
a service to cost, so I don't know whether someone would think the price
to be a bargain or a rip-off.  Anyway, I can provide cost estimates to 
organizations having a serious interest.  You may not need the services
of another organization to use the ACES; we think it's now pretty well
documented and fairly user-friendly.  If you're concerned about compiler
selection (actually, implementation selection, including hardware, etc.),
I encourage you to check out the ACES.  It's available to one and all
by anonymous FTP (sw-eng.falls-church.va.us:public/AdaIC/testing/aces/v2.1)
or via the Web (http://sw-eng.falls-church.va.us/AdaIC/testing/aces/).


Phil Brashear
CTA INCORPORATED
5100 Springfield Pike, Suite 100
Dayton, OH  45431
Voice:     (513) 258-0831
Facsimile: (513) 252-3680
brashear@sw-eng.falls-church.va.us
brashear@smtplink.cta.com
brashepw@msrc.wpafb.af.mil
brashear@saber.udayton.edu

Re: The Ada Compiler Evaluation System

[Ken Garlington]

Philip Brashear wrote:

> ...as one who has
> led both ACVC (validation suite) and ACES (evaluation suite) development
> efforts and the application thereof to various Ada compilers, I can state
> that these are the interpretations that have gone into the efforts.  If
> there is a need to change the direction of either, then there should be a
> discussion along the lines of "I suggest that the validation (or evaluation)
> effort be modified as follows:".

I suggest that the validation (or evaluation) effort be modified as follows:

1. The ACVC should be modified to improve the quality of Ada compilers
across all vendors. (examples given below, and in previous posts).

2. Alternately, something outside the ACVC should be introduced or modified
to improve the quality of Ada compilers across all vendors.

> HOWEVER (he said, finally getting to his point), both the ACVC and the
> ACES are extremely useful.

Why? Is it because...

> ACVC usage by vendors is pretty much required (at least for those selling
> to the U.S. Government)

Being required is good, so long as we can state the extent to which the ACVC
adds to vendor quality. So far, the only proposal is that, because of the
ACVC, Ada compiler quality may be better than Pascal compiler quality. That
doesn't seem to be a particularly valuable measure (e.g. as a basis for
continuing process improvement).

Of course, this appears to be the only standard of quality (and, clearly, there
is disagreement as to whether this is even _a_ standard of quality) that all
compiler vendors must meet. Is it the most efficient standard? Should there be
other standards?

> ACES usage isn't.

Neither is ISO 9001. ISO 12207. SEI CMM. The NPL tool. The Rational TestMate tool.
There's a plethora of tools and techniques that _can_ be used. They all have one
thing in common. There's nothing that _requires_ their use. Each user has to
apply (and reapply, and reapply) some or all of these requirements, and pay for
the privilege. As a result, none of these techniques apply "across all vendors,"
and so they don't meet the criteria of my questions (now suggestions).

Frankly, I had hoped this thread would turn out to be a discussion of _which_
techniques _should_ be proposed as a common basis for compiler quality measurement
and improvement. Instead, it has wormed around the tired refrains of "ACVC is defined 
to be x, so it can't be anything else" and "it's too hard to do anything" and "my 
company does good (but unspecified) stuff" and "ACVC is getting better (with some 
unspecified effect on compiler quality)".

> Some vendors are known to use the ACES; perhaps all do.  There are
> organization (such as mine) who are prepared to perform ACES evaluations
> and comparisons on a fee basis.

It is true that one answer to these suggestions is: "Each user has to pay to
get whatever quality he wants." However, this process seems very inefficient.
For example, once one user requests an evaluation, do you make that
evaluation available free of charge to subsequent users? If not, then
users are having to pay for work that's already done. If so, then I pity
the poor user who had to pay for the rest of the user base (in particular.
since on more than one occasion _I've_ been that user!)

Somehow, it makes sense for one organization to do NIST certification for
all users, but if you want (other) measures of quality, it doesn't make sense for
one organization to do those measures for all users, even though there are now
some fairly widely-accepted common measures of software quality.

If it's OK for users to pay for quality improvements, why not have users
pay for AVO support? If they want NIST certification, fine. If not, they can use
the money to pay for ISO 9001 certification, or more tool functionality, or
something else. It appears to be the stock answer for every (other) potential
standard method of measuring/improving compiler quality. Why not apply it to ACVC?

-- 
LMTAS - "Our Brand Means Quality"

Re: The Ada Compiler Evaluation System

[Robert Dewar]

Ken Garlington said

  >If it's OK for users to pay for quality improvements, why not have
  >users pay for AVO support? If they want NIST certification, fine. If
  >not, they can use something else. It appears to be the stock answer
  >for every (other) potential standard method of measuring/improving
  >compiler quality. Why not apply it to ACVC?

I think you answered your own question. The point is really quite simple.
(Again I remind you we are talking here only about internal DoD policy).
The DoD should require of all Ada vendors only those quality criteria
which arguably apply in a cost-effective manner across the board to all
DoD users of Ada.

Why should we do even that much? That's actually not entirely clear. If
all procurement officers in the DoD were to require all vendors to do
XXX for their project in order to sell their compiler, that would in
\apractice have the same effect as a mandate, i.e. vendors who want to
sell to the DoD would have to do XXX. Of course, as is the case even
with the "requirement" of validation, if you don't want to sell to the
DoD, you can ignore even validation.

I guess the argument in favor of doing that much is not so much to control
vendors as to control procurement officers. If the DoD has determined that
for it purposes all Ada compilers should satisfy XXX, then presumably it
has an interest in mandating XXX and taking away the freedom of procurement
officers to say "I don't care about XXX, I would rather save money." That
seems to be the motivation behind requiring validation. The requirement for
validaton does of course increase compiler costs (e.g. in comparison with
C++) but if the DoD determines that this cost increase is commensurate
to the quality improvement, then the general requirement makes sense
(although in that case, DoD should not complain that Ada is more expensive
than C++ to the extent that this increase in cost is caused by the
extra requirement in the Ada case -- in practice I think this inrease
in cost is fairly minimal, yes validation is expensive, but the
expense is not huge in proportion to other factors -- talking from
one vendors point of view anyway).

Going back to the original issue, Why not do more? i.e. why not have DoD
universally require more than ACVC testing. There are two subquestions:

  1) would requiring more increase quality for all possible applications
      of Ada in the DoD?

  2) even if the answer to 1) is no, should it be done anyway?

Let's take 1) first. If the answer is yes, the DoD still needs to do
an assessment of whether it would increase the cost unacceptably. This
is certainly difficult to do assess across the board. The current statement
of policy is a judgment that either the answer to 1) is no, or that if
it is yes, then the cost would be too high.

The answer to 2) is almost certainly no. The issue is that if you mandate
extra testing or certification that does NOT help quality for certain
applications, it will most certainly DECREASE quality for these applications
by diverting resources.

It's quite interesting that no one else has hopped into this thread.
Certainly I assume that all the vendors and users reading this thread
are interested in improving quality.

  Q. Are you interested in improved quality
  
  A. Of course.
  
  Q. Do you think all compiler vendors should be required by DoD to do XXX?
  
  A. Hmmm! not so simple to answer, we would have to look at XXX very closely.

Ken worries that if every user does their own evaluation that is inefficient.
Sure that is true, but no centralized DoD testing is going to relieve you
of that requirement. Early on some naive people thought that validation
was some marvellous panacea that would relieve them of the need to
carefully evaluate competing compiler products. What Ken seems to be
looking for at times is some other formula that would provide that
panacea, and I am afraid it is not possible. Even if all sorts of
extra requirements were placed, it would not relieve someone starting
out on a major project from the need to evaluate to see if the tools
to be used met the needs.

On the other hand, you certainly don't have to do stuff yourself. You
can make the vendor do it. No one suggests that every user do their
own ACES testing.

Let's suppose that you have determined that ACES testing is useful. In
that case you should simply only look at products for which ACES testing
results are available (this has certainly been done in some cases).
Similarly, if you think ISO 9000 certification is useful, then by all
means require it (other users of Ada have done so, Alsys did not get
ISO 9000 certification for its own internal requrements, but because
customers required it).

Ken, in the case of your project, were extra requirements like this
set down? If not why not?

In the commercial world, there is no central body making requirements
on, e.g. C or COBOL compilers, with regard to testing requirements.
Individual users can and do make decisions to only acquire copilers
that meet certain requirements -- for example it is common to require
NIST certification for C and COBOL, common enough that in practice
almost all C and COBOL compilers are usually NIST certified. However
that particular market place (C and COBOL compilers) does not seem
to have any general agreemnt on other testing methods.

Ken, what's so terrible about letting the market place decide what
requirements to place on tools that are acquired, instead of concentrating
on having DoD make general requirements that might or might not be suitable
for your particular project.

Yes I know, you keep asking why, if this argument is valid, should it
not be applied to the ACVC itself? The answer is that there is a general
consensus in the DoD that this is a good idea. Furthermore, it is not
unusual historically. The GSA very early on required the equivalent of
NIST testing for all COBOL compilers. The basic idea is that the ACVC
measures conformance to the standard. The DoD requirement is that you
use an Ada compiler in certain cicumtances. The ACVC testing essentially
defines what constitutes an Ada compiler. There is no mandate to use
Ada compilers with certain characteristics, "quality" or otherwise.
Ken wants the mandate strengthened in this area, but it is not clear
there is any consensus for such strengthening.

P.S. Ken, you recommend improving the quality of the ACVC suite. Fine, but
can we have some specific constructive suggestions? The ACVC development
effort is quite open, with intermediate results available for public
review. There is a review committee that represents vendors, language
designers and users that evaluates the project on an ongoing basis
and is open to suggestions from the entire community.

It maybe that Ken knows better than other people on the committee what
should go into the ACVC tests and should have been chosen as a committee
member instead of me or some other member of the committee. I would be
more convinced of that possibility if I saw specific suggestions from
Ken based on careful examination of the existing 2.1 suite as it
develops. In fact I rather suspect that Ken's reaction to the ACVC
is based entirely on version 1.11, which is at this stage very out of
date, both in terms of language features covered, and testing philosophy.
suggestions have bee

Re: The Ada Compiler Evaluation System

[Ken Garlington]

Robert Dewar wrote:
> 
> I think you answered your own question. The point is really quite simple.
> (Again I remind you we are talking here only about internal DoD policy).
> The DoD should require of all Ada vendors only those quality criteria
> which arguably apply in a cost-effective manner across the board to all
> DoD users of Ada.

OK. Given that" [t]he DoD should require of all Ada vendors only those
quality criteria which arguably apply in a cost-effective manner across the
board to all DoD users of Ada," what should be included in that criteria?
Should it be the ACVC? Should it be a different ACVC? Should it be something
in addition to the ACVC? Should it be something instead of the ACVC?

(Deja vu!)

> Why should we do even that much? That's actually not entirely clear.

I agree. Is there anyone that can clarify this?

> If
> all procurement officers in the DoD were to require all vendors to do
> XXX for their project in order to sell their compiler, that would in
> \apractice have the same effect as a mandate, i.e. vendors who want to
> sell to the DoD would have to do XXX. Of course, as is the case even
> with the "requirement" of validation, if you don't want to sell to the
> DoD, you can ignore even validation.

And if all users required all vendors to do XXX for their project,
regardless of what the procurement officers requested, then that also
would be the same as a mandate. It would be stronger than a DoD mandate,
in fact, since it would include commercial users.

Are there measures of compiler quality that all users should require?

> I guess the argument in favor of doing that much is not so much to control
> vendors as to control procurement officers. If the DoD has determined that
> for it purposes all Ada compilers should satisfy XXX, then presumably it
> has an interest in mandating XXX and taking away the freedom of procurement
> officers to say "I don't care about XXX, I would rather save money." That
> seems to be the motivation behind requiring validation.

That's an interesting view of the situation, and you may well be right
that the DoD requires a common measure to control procurements. However,
it does beg the question, "Why ACVC as the best/only measure?"

> The requirement for
> validaton does of course increase compiler costs (e.g. in comparison with
> C++) but if the DoD determines that this cost increase is commensurate
> to the quality improvement, then the general requirement makes sense

I absolute agree with this statement. Can we now talk about the "if" embedded
within it? Does the DoD have information that validates the ACVC's role in
quality improvement? Does anyone outside DoD have this information?

> in practice I think this inrease
> in cost is fairly minimal, yes validation is expensive, but the
> expense is not huge in proportion to other factors -- talking from
> one vendors point of view anyway).

OK, but to compute return on investment, we need more than the cost. We need
the benefit. The cost may not be huge, but a more interesting question is,
"Is it justified?"

Alternately, there may be measures that are more expensive, but in fact
bring much more quality. In the case of SEI or ISO 9001 measures, there might
be even more to the story. There might be a higher up-front cost, but a lower
cost (compared to ACVC) over the longer term.

> Going back to the original issue, Why not do more? i.e. why not have DoD
> universally require more than ACVC testing. There are two subquestions:
> 
>   1) would requiring more increase quality for all possible applications
>       of Ada in the DoD?
> 
>   2) even if the answer to 1) is no, should it be done anyway?
> 
> Let's take 1) first. If the answer is yes, the DoD still needs to do
> an assessment of whether it would increase the cost unacceptably. This
> is certainly difficult to do assess across the board. The current statement
> of policy is a judgment that either the answer to 1) is no, or that if
> it is yes, then the cost would be too high.

I think the current statement of policy might _imply_ such a judgement, but
I don't see that it states such a judgement. There is an alternaive explanation:
Maybe no one in DoD has asked these questions, or had the time to answer them.
Maybe they require NIST certification because they always require certification.

> The answer to 2) is almost certainly no. The issue is that if you mandate
> extra testing or certification that does NOT help quality for certain
> applications, it will most certainly DECREASE quality for these applications
> by diverting resources.

Granted. However, the converse -- extra testing that increases quality sufficiently
for a good ROI, or just a better ROI than ACVC -- would in fact improve resource
allocation, right?

> It's quite interesting that no one else has hopped into this thread.
> Certainly I assume that all the vendors and users reading this thread
> are interested in improving quality.

I agree. I really expected a broad sampling of opinions on this topic.

>   Q. Are you interested in improved quality
> 
>   A. Of course.
> 
>   Q. Do you think all compiler vendors should be required by DoD to do XXX?
> 
>   A. Hmmm! not so simple to answer, we would have to look at XXX very closely.

Absolutely - it's a difficult question to answer. That doesn't mean it shouldn't
be asked, however.

> Ken worries that if every user does their own evaluation that is inefficient.
> Sure that is true, but no centralized DoD testing is going to relieve you
> of that requirement.

I also agree with this. However, it might allow the users to focus more on
domain-specific tests, and less on tests that could be done globally. Right now,
we do a lot of doamin-specific testing, which means we don't have the resources
to do a lot of generic testing. I would argue both kinds of testing are important.

> Early on some naive people thought that validation
> was some marvellous panacea that would relieve them of the need to
> carefully evaluate competing compiler products. What Ken seems to be
> looking for at times is some other formula that would provide that
> panacea, and I am afraid it is not possible.

Not at all. I just want to maximize the ROI for mandated, global testing.
It may not be possible to have an effective general-purpose measure of
compiler quality beyond the ACVC. It may also be true that the ACVC provides
an excellent ROI. However, I think there is a lot of evidence in the
commericial software world that indicates effective global measures are
possible.

> Even if all sorts of
> extra requirements were placed, it would not relieve someone starting
> out on a major project from the need to evaluate to see if the tools
> to be used met the needs.

No argument there. However, it would be nice to pick up a report containing
a list of common compiler measures, rather than doing those measures yourself,
and then using that as a basis _in combination with_ your domain-specific tests.
This certainly happens in the hardware world, and in some software domains today.

> On the other hand, you certainly don't have to do stuff yourself. You
> can make the vendor do it. No one suggests that every user do their
> own ACES testing.

> Let's suppose that you have determined that ACES testing is useful. In
> that case you should simply only look at products for which ACES testing
> results are available (this has certainly been done in some cases).

Are ACES results kept in the public domain? Has ACES been performed on most Ada
compilers? Can the results be interpreted in a way that has value for most
Ada users? If the answer to all three is yes, then I would say ACES is an
example of the very thing I'm looking for.

> Similarly, if you think ISO 9000 certification is useful, then by all
> means require it (other users of Ada have done so, Alsys did not get
> ISO 9000 certification for its own internal requrements, but because
> customers required it).

OK, I will! :)

However, there's a couple of problems: First, If I set a bar on compiler quality
at the level of ISO 9001, not all vendors are interested in reaching for
that bar regardless of how much I'm willing to invest (as far as I can
tell). This drastically limits my choices, in a market where I don't have
a large number of choices to begin with (depending upon the host/target
sets I need to support). This adds risk to my project.

The second problem is that the Ada market is much smaller than the market
for languages such as C or C++. A large player in the market, if he requests
a measure that is not value added, can really do serious damage to a vendor.
I know. I have made this mistake. However, some vendors seem reluctant to discuss
whether or not a user requirement makes sense, particularly when there's
a lot of money riding on the sale. So, from my perspective, it makes more
sense to ask these questions outside of a procurement, and try to get honest
feedback from relatively disinterested parties. That way, when I go into a
procurement and require ISO 9001, SEI III, or whatever, I have confidence that
this makes sense to the vendor commercially and has a good ROI with respect
to compiler quality.

> Ken, in the case of your project, were extra requirements like this
> set down? If not why not?

Yes, we have a spec for every project. Some of these extra requirements are
now sold commercially. One of these requirements was for my vendor to
implement the equivalent of pragma Reviewable. Now, this is a standard part
of the Ada 95 language, since enough users apparently thought this function was
needed to include it in the standard (although part of an appendix). I'm looking
for the same concept, but for that which can't be put in the Ada
standard: measures of compiler quality

Let me digress briefly into how my group buys compilers:

1. We write a spec that describes what we want. It talks about features we need,
capacity limits, performance, etc. It hasn't historically talked much about
robustness, etc. since we haven't in the past had ways to measure such things.

2. We create benchmarks. Since coding hasn't started yet, we use samples of
code from the previous project, which we try to get from each domain that will
be using the compiler.

3. We release RFPs that include the spec, the SOW, and all the good contractual
language.

4. We evaluate the potential vendors, by checking their proposals against the spec,
running the benchmarks, and so forth.

5. We select a compiler, get it in-house, integrate it with the rest of the
software engineering environment, and so on.

The problem is threefold:

1. The specs and benchmarks have to be written very early in the program, before
the users know enough about their requirements to say for certainty what they
think they need. (In addition, they are busy doing trade studies and so forth, and
so don't have time to spend thinking about coding!)

2. Our requirements are always based on the last program. The next program is
always different.

3. Until recently, there weren't any commonly used measures of software quality,
so our specs never addressed them.

As a result, it occurs to me that it would be far smarter to de-emphasize the specs
and benchmarks (but not eliminate them), and to emphasize general measures of
software quality. Presumably, if a vendor has a high-quality product and process,
then (a) the users will be happier and (v) the vendor will be more likely to cope
with the inevitable changes the users will require, either those that are defined
in the spec or those that come up later.

It also occurs to me that, since these measures will be general (not tied to a
specific domain), and if users other than myself wanted them, I might not have
to bear the full cost for them. Even better, what if some other user or set of
users (including the DoD) has already paid for those measures, or could be convinced
to pay for those measures, for a large set of the available compilers? Then I would
really be smart to add those measures to my spec, since I could get them for little
or no cost!

Of course, to know which measures make sense, I would have to ask compiler vendors
their opinion. If only one vendor thought a particular measure made sense, then
I might be locking myself into that one vendor, which is a risk. On the other
hand, if most vendors seemed to think ISO 9001 (for example) was reasonable, and the
only reason they hadn't done it was a lack of funds, then at least I had somewhere to
start.

> Ken, what's so terrible about letting the market place decide what
> requirements to place on tools that are acquired, instead of concentrating
> on having DoD make general requirements that might or might not be suitable
> for your particular project.

I don't limit this discussion to what DoD should or shouldn't do. I am
perfectly willing to make this a market-driven, unofficial standard.

So, what should users be asking for from their vendors, that would be a common
measure?

> Yes I know, you keep asking why, if this argument is valid, should it
> not be applied to the ACVC itself? The answer is that there is a general
> consensus in the DoD that this is a good idea. Furthermore, it is not
> unusual historically.

There was also a general consensus in the DoD on using MIL-STDs instead of
equilalent commercial standards, and it wasn't unusual historically, either.
I suspect the trend toward acquisition reform started with one person
asking the dreaded question, "Why do we do it this way?" The first time the
question was asked, they probably got the same answer you just gave. Frankly,
it's a crappy answer from a quality perspective!

> Ken wants the mandate strengthened in this area, but it is not clear
> there is any consensus for such strengthening.

It's not clear to me, either. That's why I'm asking these questions. How
else would we discover if there was such a consensus?

> P.S. Ken, you recommend improving the quality of the ACVC suite. Fine, but
> can we have some specific constructive suggestions?

Sure. Change the scope of ACVC as follows:

  1. Require vendors to attach their software development plans to their
     ACVC test reports, along with a signed statement that the product was
     developed using the plan.

  2. Require vendors to attach their internal software test procedures to their
     ACVC test reports, along with a signed statement that the product passed
     those tests.

  3. Require vendors to provide updates to the ACVC based on their regression
     test suites, where those tests (a) can be made public and (b) can be
     written as a general-pupose test.

  4. Require vendors to sign a statement that they conform to ISO 9001 and/or
     SEI III.

-- 
LMTAS - "Our Brand Means Quality"

Re: The Ada Compiler Evaluation System

[Robert Dewar]

"OK. Given that" [t]he DoD should require of all Ada vendors only those
quality criteria which arguably apply in a cost-effective manner across the
board to all DoD users of Ada," what should be included in that criteria?
Should it be the ACVC? Should it be a different ACVC? Should it be something
in addition to the ACVC? Should it be something instead of the ACVC?"

A starting point would be to study what has already been decided in this
area by the project team and advisory committee for the new ACVC test
suite. In particular, I would take a little time to study the ACVC suite,
both old and new.

Re: The Ada Compiler Evaluation System

[Ken Garlington]

Robert Dewar wrote:
> 
> A starting point would be to study what has already been decided in this
> area by the project team and advisory committee for the new ACVC test
> suite. In particular, I would take a little time to study the ACVC suite,
> both old and new.

Been there, done that. Even after I quoted excerpts from the ACVC 2.0
documentation, the response has been to RTFM and to "stop bashing those
poor ACVC folks."

No possible benefit is possible from this conversation. I withdraw. Good bye.

-- 
LMTAS - "Our Brand Means Quality"

Re: The Ada Compiler Evaluation System

[Robert Dewar]

Ken said

"Been there, done that. Even after I quoted excerpts from the ACVC 2.0
documentation, the response has been to RTFM and to "stop bashing those
poor ACVC folks.""

Ken, we are not talking about the "ACVC 2.0 documentation" we are 
suggesting that you look at the tests themselves. If you have "been there,
done that", then I would really be interested in your opinion of the
new testing approach (and so would the ACVC test development team!)

P.S. on another matter, I think the important distinction that Gary was
trying to make was that you can pass or fail the ACVC tests, they are
obvjective in that sense. You cannot pass or fail the ACEC tests -- any
more than you can pass or fail the SpecMark (which incidentally is
a TERRIBLY misleading indication of microprocessor performance, as
everyone in the industry knows, but everyone keeps advertising it,
because users do indeed depend on it -- this is one of the well known
problems with standard performance tests).

You also said that you thought all compilers should be run through the
ACEC tests. That's certainly reasonable if the market requires it. It
might even be reasonable for the DoD to fund this. It certainly *is*
reasonable, and very common, for customers to require ACEC testing,
and they don't "each pay for the" tests, the vendor runs them once
(why would you think things work differently).

Of course some vendors might not be inerested in the ACEC tests. We never
bothered for instance to run Ada/Ed through the ACEC tests, except as
basic functionality tests, since the performance of Ada/Ed was not
particularly relevant to its users, or, more accurately, maybe it
was relevant, but performance was *not* a salient feature of Ada/Ed.
Other vendors may simply not be interested in customers who require
ACES testing, and that's quite a reasonable position (they won't
be able to sell to these customers). 

You can't even force all vendors to validate, let alone run other tests
in any compleely general way. You *can* and *should* request vendros
to meet your needs, and requesting ACEC test results is certainly
reasonable (it is a bit of a surprise to me that the F22 program did
not require ACEC testing).

As for publishing test results, as far as I know ACEC test results can
be published -- this is not true of all test suites, there are often
restrictions. Spec results can only be published in a certain manner,
and as far as I know results from the NPL stress tests cannot be
published at all (although this might have changed) -- basically
the idea behind such restrictions is to avoid over-focus on the
results of a particular suite, or to avoid its use in advertising.
That decision is of course up to the vendor of the test suite.

Re: The Ada Compiler Evaluation System

[Laurent Guerby]

Robert Dewar writes
[deleted]
: P.S. on another matter, I think the important distinction that Gary was
                                                                 ^^^^

   Hum, I don't think Gary is posting here ;-).

: trying to make was that you can pass or fail the ACVC tests, they are
: obvjective in that sense. You cannot pass or fail the ACEC tests -- any
                                                        ^^^^
: more than you can pass or fail the SpecMark (which incidentally is
: a TERRIBLY misleading indication of microprocessor performance, as
: everyone in the industry knows, but everyone keeps advertising it,
: because users do indeed depend on it -- this is one of the well known
: problems with standard performance tests).
[deleted]

   I think the new name is ACES (fusion ?). 

   Another thing about evaluation  (performance) is the  configuration
problem. It's hard to guess from synthetic benchmarks what will be the
performance   of your   application       on your site     with   your
configuration. The only way to  have a reliable  answer is to test  it
yourself on your site (of course there's a price for it).

-- 
--  Laurent Guerby, student at Telecom Bretagne (France), Team Ada.
--  "Use the Source, Luke. The Source will be with you, always (GPL)."
--  http://www-eleves.enst-bretagne.fr/~guerby/ (GATO Project).
--  Try GNAT, the GNU Ada 95 compiler (ftp://cs.nyu.edu/pub/gnat).