* The Ada Compiler Evaluation System @ 1996-04-17 0:00 Philip Brashear 1996-04-18 0:00 ` Ken Garlington 1996-04-29 0:00 ` Laurent Guerby 0 siblings, 2 replies; 8+ messages in thread From: Philip Brashear @ 1996-04-17 0:00 UTC (permalink / raw) There has been a rather acrimonious discussion of compiler validation and evaluation recently, and I really don't understand the bitterness. Those of us in the (?quasi?) official Ada compiler testing business understand "validation" to mean testing against the standard (insofar as that is possible), simply to get _some_ assurance that the compiler implements the standard correctly. We know that we cannot _ensure_ that any compiler does so. On the other hand, we understand "evaluation" to mean some attempt to determine whether a compiler meets the performance and usability needs of a particular project. (We _assume_ that the compiler implements the standard correctly.) Now, I don't claim that these two notions are completely defined, nor that "our" interpretations are the only reasonable ones. However, as one who has led both ACVC (validation suite) and ACES (evaluation suite) development efforts and the application thereof to various Ada compilers, I can state that these are the interpretations that have gone into the efforts. If there is a need to change the direction of either, then there should be a discussion along the lines of "I suggest that the validation (or evaluation) effort be modified as follows:". Unfortunately, it is very likely that no more DoD funding will be applied to the evaluation effort, and the funding available for maintenance/upgrade of the validation suite (and process) will be quite limited. HOWEVER (he said, finally getting to his point), both the ACVC and the ACES are extremely useful. ACVC usage by vendors is pretty much required (at least for those selling to the U.S. Government); ACES usage isn't. Some vendors are known to use the ACES; perhaps all do. There are organization (such as mine) who are prepared to perform ACES evaluations and comparisons on a fee basis. I don't know what one would _expect_ such a service to cost, so I don't know whether someone would think the price to be a bargain or a rip-off. Anyway, I can provide cost estimates to organizations having a serious interest. You may not need the services of another organization to use the ACES; we think it's now pretty well documented and fairly user-friendly. If you're concerned about compiler selection (actually, implementation selection, including hardware, etc.), I encourage you to check out the ACES. It's available to one and all by anonymous FTP (sw-eng.falls-church.va.us:public/AdaIC/testing/aces/v2.1) or via the Web (http://sw-eng.falls-church.va.us/AdaIC/testing/aces/). Phil Brashear CTA INCORPORATED 5100 Springfield Pike, Suite 100 Dayton, OH 45431 Voice: (513) 258-0831 Facsimile: (513) 252-3680 brashear@sw-eng.falls-church.va.us brashear@smtplink.cta.com brashepw@msrc.wpafb.af.mil brashear@saber.udayton.edu ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-17 0:00 The Ada Compiler Evaluation System Philip Brashear @ 1996-04-18 0:00 ` Ken Garlington 1996-04-21 0:00 ` Robert Dewar 1996-04-29 0:00 ` Laurent Guerby 1 sibling, 1 reply; 8+ messages in thread From: Ken Garlington @ 1996-04-18 0:00 UTC (permalink / raw) Philip Brashear wrote: > ...as one who has > led both ACVC (validation suite) and ACES (evaluation suite) development > efforts and the application thereof to various Ada compilers, I can state > that these are the interpretations that have gone into the efforts. If > there is a need to change the direction of either, then there should be a > discussion along the lines of "I suggest that the validation (or evaluation) > effort be modified as follows:". I suggest that the validation (or evaluation) effort be modified as follows: 1. The ACVC should be modified to improve the quality of Ada compilers across all vendors. (examples given below, and in previous posts). 2. Alternately, something outside the ACVC should be introduced or modified to improve the quality of Ada compilers across all vendors. > HOWEVER (he said, finally getting to his point), both the ACVC and the > ACES are extremely useful. Why? Is it because... > ACVC usage by vendors is pretty much required (at least for those selling > to the U.S. Government) Being required is good, so long as we can state the extent to which the ACVC adds to vendor quality. So far, the only proposal is that, because of the ACVC, Ada compiler quality may be better than Pascal compiler quality. That doesn't seem to be a particularly valuable measure (e.g. as a basis for continuing process improvement). Of course, this appears to be the only standard of quality (and, clearly, there is disagreement as to whether this is even _a_ standard of quality) that all compiler vendors must meet. Is it the most efficient standard? Should there be other standards? > ACES usage isn't. Neither is ISO 9001. ISO 12207. SEI CMM. The NPL tool. The Rational TestMate tool. There's a plethora of tools and techniques that _can_ be used. They all have one thing in common. There's nothing that _requires_ their use. Each user has to apply (and reapply, and reapply) some or all of these requirements, and pay for the privilege. As a result, none of these techniques apply "across all vendors," and so they don't meet the criteria of my questions (now suggestions). Frankly, I had hoped this thread would turn out to be a discussion of _which_ techniques _should_ be proposed as a common basis for compiler quality measurement and improvement. Instead, it has wormed around the tired refrains of "ACVC is defined to be x, so it can't be anything else" and "it's too hard to do anything" and "my company does good (but unspecified) stuff" and "ACVC is getting better (with some unspecified effect on compiler quality)". > Some vendors are known to use the ACES; perhaps all do. There are > organization (such as mine) who are prepared to perform ACES evaluations > and comparisons on a fee basis. It is true that one answer to these suggestions is: "Each user has to pay to get whatever quality he wants." However, this process seems very inefficient. For example, once one user requests an evaluation, do you make that evaluation available free of charge to subsequent users? If not, then users are having to pay for work that's already done. If so, then I pity the poor user who had to pay for the rest of the user base (in particular. since on more than one occasion _I've_ been that user!) Somehow, it makes sense for one organization to do NIST certification for all users, but if you want (other) measures of quality, it doesn't make sense for one organization to do those measures for all users, even though there are now some fairly widely-accepted common measures of software quality. If it's OK for users to pay for quality improvements, why not have users pay for AVO support? If they want NIST certification, fine. If not, they can use the money to pay for ISO 9001 certification, or more tool functionality, or something else. It appears to be the stock answer for every (other) potential standard method of measuring/improving compiler quality. Why not apply it to ACVC? -- LMTAS - "Our Brand Means Quality" ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-18 0:00 ` Ken Garlington @ 1996-04-21 0:00 ` Robert Dewar 1996-04-22 0:00 ` Ken Garlington 0 siblings, 1 reply; 8+ messages in thread From: Robert Dewar @ 1996-04-21 0:00 UTC (permalink / raw) Ken Garlington said >If it's OK for users to pay for quality improvements, why not have >users pay for AVO support? If they want NIST certification, fine. If >not, they can use something else. It appears to be the stock answer >for every (other) potential standard method of measuring/improving >compiler quality. Why not apply it to ACVC? I think you answered your own question. The point is really quite simple. (Again I remind you we are talking here only about internal DoD policy). The DoD should require of all Ada vendors only those quality criteria which arguably apply in a cost-effective manner across the board to all DoD users of Ada. Why should we do even that much? That's actually not entirely clear. If all procurement officers in the DoD were to require all vendors to do XXX for their project in order to sell their compiler, that would in \apractice have the same effect as a mandate, i.e. vendors who want to sell to the DoD would have to do XXX. Of course, as is the case even with the "requirement" of validation, if you don't want to sell to the DoD, you can ignore even validation. I guess the argument in favor of doing that much is not so much to control vendors as to control procurement officers. If the DoD has determined that for it purposes all Ada compilers should satisfy XXX, then presumably it has an interest in mandating XXX and taking away the freedom of procurement officers to say "I don't care about XXX, I would rather save money." That seems to be the motivation behind requiring validation. The requirement for validaton does of course increase compiler costs (e.g. in comparison with C++) but if the DoD determines that this cost increase is commensurate to the quality improvement, then the general requirement makes sense (although in that case, DoD should not complain that Ada is more expensive than C++ to the extent that this increase in cost is caused by the extra requirement in the Ada case -- in practice I think this inrease in cost is fairly minimal, yes validation is expensive, but the expense is not huge in proportion to other factors -- talking from one vendors point of view anyway). Going back to the original issue, Why not do more? i.e. why not have DoD universally require more than ACVC testing. There are two subquestions: 1) would requiring more increase quality for all possible applications of Ada in the DoD? 2) even if the answer to 1) is no, should it be done anyway? Let's take 1) first. If the answer is yes, the DoD still needs to do an assessment of whether it would increase the cost unacceptably. This is certainly difficult to do assess across the board. The current statement of policy is a judgment that either the answer to 1) is no, or that if it is yes, then the cost would be too high. The answer to 2) is almost certainly no. The issue is that if you mandate extra testing or certification that does NOT help quality for certain applications, it will most certainly DECREASE quality for these applications by diverting resources. It's quite interesting that no one else has hopped into this thread. Certainly I assume that all the vendors and users reading this thread are interested in improving quality. Q. Are you interested in improved quality A. Of course. Q. Do you think all compiler vendors should be required by DoD to do XXX? A. Hmmm! not so simple to answer, we would have to look at XXX very closely. Ken worries that if every user does their own evaluation that is inefficient. Sure that is true, but no centralized DoD testing is going to relieve you of that requirement. Early on some naive people thought that validation was some marvellous panacea that would relieve them of the need to carefully evaluate competing compiler products. What Ken seems to be looking for at times is some other formula that would provide that panacea, and I am afraid it is not possible. Even if all sorts of extra requirements were placed, it would not relieve someone starting out on a major project from the need to evaluate to see if the tools to be used met the needs. On the other hand, you certainly don't have to do stuff yourself. You can make the vendor do it. No one suggests that every user do their own ACES testing. Let's suppose that you have determined that ACES testing is useful. In that case you should simply only look at products for which ACES testing results are available (this has certainly been done in some cases). Similarly, if you think ISO 9000 certification is useful, then by all means require it (other users of Ada have done so, Alsys did not get ISO 9000 certification for its own internal requrements, but because customers required it). Ken, in the case of your project, were extra requirements like this set down? If not why not? In the commercial world, there is no central body making requirements on, e.g. C or COBOL compilers, with regard to testing requirements. Individual users can and do make decisions to only acquire copilers that meet certain requirements -- for example it is common to require NIST certification for C and COBOL, common enough that in practice almost all C and COBOL compilers are usually NIST certified. However that particular market place (C and COBOL compilers) does not seem to have any general agreemnt on other testing methods. Ken, what's so terrible about letting the market place decide what requirements to place on tools that are acquired, instead of concentrating on having DoD make general requirements that might or might not be suitable for your particular project. Yes I know, you keep asking why, if this argument is valid, should it not be applied to the ACVC itself? The answer is that there is a general consensus in the DoD that this is a good idea. Furthermore, it is not unusual historically. The GSA very early on required the equivalent of NIST testing for all COBOL compilers. The basic idea is that the ACVC measures conformance to the standard. The DoD requirement is that you use an Ada compiler in certain cicumtances. The ACVC testing essentially defines what constitutes an Ada compiler. There is no mandate to use Ada compilers with certain characteristics, "quality" or otherwise. Ken wants the mandate strengthened in this area, but it is not clear there is any consensus for such strengthening. P.S. Ken, you recommend improving the quality of the ACVC suite. Fine, but can we have some specific constructive suggestions? The ACVC development effort is quite open, with intermediate results available for public review. There is a review committee that represents vendors, language designers and users that evaluates the project on an ongoing basis and is open to suggestions from the entire community. It maybe that Ken knows better than other people on the committee what should go into the ACVC tests and should have been chosen as a committee member instead of me or some other member of the committee. I would be more convinced of that possibility if I saw specific suggestions from Ken based on careful examination of the existing 2.1 suite as it develops. In fact I rather suspect that Ken's reaction to the ACVC is based entirely on version 1.11, which is at this stage very out of date, both in terms of language features covered, and testing philosophy. suggestions have bee ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-21 0:00 ` Robert Dewar @ 1996-04-22 0:00 ` Ken Garlington 1996-04-24 0:00 ` Robert Dewar 0 siblings, 1 reply; 8+ messages in thread From: Ken Garlington @ 1996-04-22 0:00 UTC (permalink / raw) Robert Dewar wrote: > > I think you answered your own question. The point is really quite simple. > (Again I remind you we are talking here only about internal DoD policy). > The DoD should require of all Ada vendors only those quality criteria > which arguably apply in a cost-effective manner across the board to all > DoD users of Ada. OK. Given that" [t]he DoD should require of all Ada vendors only those quality criteria which arguably apply in a cost-effective manner across the board to all DoD users of Ada," what should be included in that criteria? Should it be the ACVC? Should it be a different ACVC? Should it be something in addition to the ACVC? Should it be something instead of the ACVC? (Deja vu!) > Why should we do even that much? That's actually not entirely clear. I agree. Is there anyone that can clarify this? > If > all procurement officers in the DoD were to require all vendors to do > XXX for their project in order to sell their compiler, that would in > \apractice have the same effect as a mandate, i.e. vendors who want to > sell to the DoD would have to do XXX. Of course, as is the case even > with the "requirement" of validation, if you don't want to sell to the > DoD, you can ignore even validation. And if all users required all vendors to do XXX for their project, regardless of what the procurement officers requested, then that also would be the same as a mandate. It would be stronger than a DoD mandate, in fact, since it would include commercial users. Are there measures of compiler quality that all users should require? > I guess the argument in favor of doing that much is not so much to control > vendors as to control procurement officers. If the DoD has determined that > for it purposes all Ada compilers should satisfy XXX, then presumably it > has an interest in mandating XXX and taking away the freedom of procurement > officers to say "I don't care about XXX, I would rather save money." That > seems to be the motivation behind requiring validation. That's an interesting view of the situation, and you may well be right that the DoD requires a common measure to control procurements. However, it does beg the question, "Why ACVC as the best/only measure?" > The requirement for > validaton does of course increase compiler costs (e.g. in comparison with > C++) but if the DoD determines that this cost increase is commensurate > to the quality improvement, then the general requirement makes sense I absolute agree with this statement. Can we now talk about the "if" embedded within it? Does the DoD have information that validates the ACVC's role in quality improvement? Does anyone outside DoD have this information? > in practice I think this inrease > in cost is fairly minimal, yes validation is expensive, but the > expense is not huge in proportion to other factors -- talking from > one vendors point of view anyway). OK, but to compute return on investment, we need more than the cost. We need the benefit. The cost may not be huge, but a more interesting question is, "Is it justified?" Alternately, there may be measures that are more expensive, but in fact bring much more quality. In the case of SEI or ISO 9001 measures, there might be even more to the story. There might be a higher up-front cost, but a lower cost (compared to ACVC) over the longer term. > Going back to the original issue, Why not do more? i.e. why not have DoD > universally require more than ACVC testing. There are two subquestions: > > 1) would requiring more increase quality for all possible applications > of Ada in the DoD? > > 2) even if the answer to 1) is no, should it be done anyway? > > Let's take 1) first. If the answer is yes, the DoD still needs to do > an assessment of whether it would increase the cost unacceptably. This > is certainly difficult to do assess across the board. The current statement > of policy is a judgment that either the answer to 1) is no, or that if > it is yes, then the cost would be too high. I think the current statement of policy might _imply_ such a judgement, but I don't see that it states such a judgement. There is an alternaive explanation: Maybe no one in DoD has asked these questions, or had the time to answer them. Maybe they require NIST certification because they always require certification. > The answer to 2) is almost certainly no. The issue is that if you mandate > extra testing or certification that does NOT help quality for certain > applications, it will most certainly DECREASE quality for these applications > by diverting resources. Granted. However, the converse -- extra testing that increases quality sufficiently for a good ROI, or just a better ROI than ACVC -- would in fact improve resource allocation, right? > It's quite interesting that no one else has hopped into this thread. > Certainly I assume that all the vendors and users reading this thread > are interested in improving quality. I agree. I really expected a broad sampling of opinions on this topic. > Q. Are you interested in improved quality > > A. Of course. > > Q. Do you think all compiler vendors should be required by DoD to do XXX? > > A. Hmmm! not so simple to answer, we would have to look at XXX very closely. Absolutely - it's a difficult question to answer. That doesn't mean it shouldn't be asked, however. > Ken worries that if every user does their own evaluation that is inefficient. > Sure that is true, but no centralized DoD testing is going to relieve you > of that requirement. I also agree with this. However, it might allow the users to focus more on domain-specific tests, and less on tests that could be done globally. Right now, we do a lot of doamin-specific testing, which means we don't have the resources to do a lot of generic testing. I would argue both kinds of testing are important. > Early on some naive people thought that validation > was some marvellous panacea that would relieve them of the need to > carefully evaluate competing compiler products. What Ken seems to be > looking for at times is some other formula that would provide that > panacea, and I am afraid it is not possible. Not at all. I just want to maximize the ROI for mandated, global testing. It may not be possible to have an effective general-purpose measure of compiler quality beyond the ACVC. It may also be true that the ACVC provides an excellent ROI. However, I think there is a lot of evidence in the commericial software world that indicates effective global measures are possible. > Even if all sorts of > extra requirements were placed, it would not relieve someone starting > out on a major project from the need to evaluate to see if the tools > to be used met the needs. No argument there. However, it would be nice to pick up a report containing a list of common compiler measures, rather than doing those measures yourself, and then using that as a basis _in combination with_ your domain-specific tests. This certainly happens in the hardware world, and in some software domains today. > On the other hand, you certainly don't have to do stuff yourself. You > can make the vendor do it. No one suggests that every user do their > own ACES testing. > Let's suppose that you have determined that ACES testing is useful. In > that case you should simply only look at products for which ACES testing > results are available (this has certainly been done in some cases). Are ACES results kept in the public domain? Has ACES been performed on most Ada compilers? Can the results be interpreted in a way that has value for most Ada users? If the answer to all three is yes, then I would say ACES is an example of the very thing I'm looking for. > Similarly, if you think ISO 9000 certification is useful, then by all > means require it (other users of Ada have done so, Alsys did not get > ISO 9000 certification for its own internal requrements, but because > customers required it). OK, I will! :) However, there's a couple of problems: First, If I set a bar on compiler quality at the level of ISO 9001, not all vendors are interested in reaching for that bar regardless of how much I'm willing to invest (as far as I can tell). This drastically limits my choices, in a market where I don't have a large number of choices to begin with (depending upon the host/target sets I need to support). This adds risk to my project. The second problem is that the Ada market is much smaller than the market for languages such as C or C++. A large player in the market, if he requests a measure that is not value added, can really do serious damage to a vendor. I know. I have made this mistake. However, some vendors seem reluctant to discuss whether or not a user requirement makes sense, particularly when there's a lot of money riding on the sale. So, from my perspective, it makes more sense to ask these questions outside of a procurement, and try to get honest feedback from relatively disinterested parties. That way, when I go into a procurement and require ISO 9001, SEI III, or whatever, I have confidence that this makes sense to the vendor commercially and has a good ROI with respect to compiler quality. > Ken, in the case of your project, were extra requirements like this > set down? If not why not? Yes, we have a spec for every project. Some of these extra requirements are now sold commercially. One of these requirements was for my vendor to implement the equivalent of pragma Reviewable. Now, this is a standard part of the Ada 95 language, since enough users apparently thought this function was needed to include it in the standard (although part of an appendix). I'm looking for the same concept, but for that which can't be put in the Ada standard: measures of compiler quality Let me digress briefly into how my group buys compilers: 1. We write a spec that describes what we want. It talks about features we need, capacity limits, performance, etc. It hasn't historically talked much about robustness, etc. since we haven't in the past had ways to measure such things. 2. We create benchmarks. Since coding hasn't started yet, we use samples of code from the previous project, which we try to get from each domain that will be using the compiler. 3. We release RFPs that include the spec, the SOW, and all the good contractual language. 4. We evaluate the potential vendors, by checking their proposals against the spec, running the benchmarks, and so forth. 5. We select a compiler, get it in-house, integrate it with the rest of the software engineering environment, and so on. The problem is threefold: 1. The specs and benchmarks have to be written very early in the program, before the users know enough about their requirements to say for certainty what they think they need. (In addition, they are busy doing trade studies and so forth, and so don't have time to spend thinking about coding!) 2. Our requirements are always based on the last program. The next program is always different. 3. Until recently, there weren't any commonly used measures of software quality, so our specs never addressed them. As a result, it occurs to me that it would be far smarter to de-emphasize the specs and benchmarks (but not eliminate them), and to emphasize general measures of software quality. Presumably, if a vendor has a high-quality product and process, then (a) the users will be happier and (v) the vendor will be more likely to cope with the inevitable changes the users will require, either those that are defined in the spec or those that come up later. It also occurs to me that, since these measures will be general (not tied to a specific domain), and if users other than myself wanted them, I might not have to bear the full cost for them. Even better, what if some other user or set of users (including the DoD) has already paid for those measures, or could be convinced to pay for those measures, for a large set of the available compilers? Then I would really be smart to add those measures to my spec, since I could get them for little or no cost! Of course, to know which measures make sense, I would have to ask compiler vendors their opinion. If only one vendor thought a particular measure made sense, then I might be locking myself into that one vendor, which is a risk. On the other hand, if most vendors seemed to think ISO 9001 (for example) was reasonable, and the only reason they hadn't done it was a lack of funds, then at least I had somewhere to start. > Ken, what's so terrible about letting the market place decide what > requirements to place on tools that are acquired, instead of concentrating > on having DoD make general requirements that might or might not be suitable > for your particular project. I don't limit this discussion to what DoD should or shouldn't do. I am perfectly willing to make this a market-driven, unofficial standard. So, what should users be asking for from their vendors, that would be a common measure? > Yes I know, you keep asking why, if this argument is valid, should it > not be applied to the ACVC itself? The answer is that there is a general > consensus in the DoD that this is a good idea. Furthermore, it is not > unusual historically. There was also a general consensus in the DoD on using MIL-STDs instead of equilalent commercial standards, and it wasn't unusual historically, either. I suspect the trend toward acquisition reform started with one person asking the dreaded question, "Why do we do it this way?" The first time the question was asked, they probably got the same answer you just gave. Frankly, it's a crappy answer from a quality perspective! > Ken wants the mandate strengthened in this area, but it is not clear > there is any consensus for such strengthening. It's not clear to me, either. That's why I'm asking these questions. How else would we discover if there was such a consensus? > P.S. Ken, you recommend improving the quality of the ACVC suite. Fine, but > can we have some specific constructive suggestions? Sure. Change the scope of ACVC as follows: 1. Require vendors to attach their software development plans to their ACVC test reports, along with a signed statement that the product was developed using the plan. 2. Require vendors to attach their internal software test procedures to their ACVC test reports, along with a signed statement that the product passed those tests. 3. Require vendors to provide updates to the ACVC based on their regression test suites, where those tests (a) can be made public and (b) can be written as a general-pupose test. 4. Require vendors to sign a statement that they conform to ISO 9001 and/or SEI III. -- LMTAS - "Our Brand Means Quality" ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-22 0:00 ` Ken Garlington @ 1996-04-24 0:00 ` Robert Dewar 1996-04-26 0:00 ` Ken Garlington 0 siblings, 1 reply; 8+ messages in thread From: Robert Dewar @ 1996-04-24 0:00 UTC (permalink / raw) "OK. Given that" [t]he DoD should require of all Ada vendors only those quality criteria which arguably apply in a cost-effective manner across the board to all DoD users of Ada," what should be included in that criteria? Should it be the ACVC? Should it be a different ACVC? Should it be something in addition to the ACVC? Should it be something instead of the ACVC?" A starting point would be to study what has already been decided in this area by the project team and advisory committee for the new ACVC test suite. In particular, I would take a little time to study the ACVC suite, both old and new. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-24 0:00 ` Robert Dewar @ 1996-04-26 0:00 ` Ken Garlington 1996-04-27 0:00 ` Robert Dewar 0 siblings, 1 reply; 8+ messages in thread From: Ken Garlington @ 1996-04-26 0:00 UTC (permalink / raw) Robert Dewar wrote: > > A starting point would be to study what has already been decided in this > area by the project team and advisory committee for the new ACVC test > suite. In particular, I would take a little time to study the ACVC suite, > both old and new. Been there, done that. Even after I quoted excerpts from the ACVC 2.0 documentation, the response has been to RTFM and to "stop bashing those poor ACVC folks." No possible benefit is possible from this conversation. I withdraw. Good bye. -- LMTAS - "Our Brand Means Quality" ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-26 0:00 ` Ken Garlington @ 1996-04-27 0:00 ` Robert Dewar 0 siblings, 0 replies; 8+ messages in thread From: Robert Dewar @ 1996-04-27 0:00 UTC (permalink / raw) Ken said "Been there, done that. Even after I quoted excerpts from the ACVC 2.0 documentation, the response has been to RTFM and to "stop bashing those poor ACVC folks."" Ken, we are not talking about the "ACVC 2.0 documentation" we are suggesting that you look at the tests themselves. If you have "been there, done that", then I would really be interested in your opinion of the new testing approach (and so would the ACVC test development team!) P.S. on another matter, I think the important distinction that Gary was trying to make was that you can pass or fail the ACVC tests, they are obvjective in that sense. You cannot pass or fail the ACEC tests -- any more than you can pass or fail the SpecMark (which incidentally is a TERRIBLY misleading indication of microprocessor performance, as everyone in the industry knows, but everyone keeps advertising it, because users do indeed depend on it -- this is one of the well known problems with standard performance tests). You also said that you thought all compilers should be run through the ACEC tests. That's certainly reasonable if the market requires it. It might even be reasonable for the DoD to fund this. It certainly *is* reasonable, and very common, for customers to require ACEC testing, and they don't "each pay for the" tests, the vendor runs them once (why would you think things work differently). Of course some vendors might not be inerested in the ACEC tests. We never bothered for instance to run Ada/Ed through the ACEC tests, except as basic functionality tests, since the performance of Ada/Ed was not particularly relevant to its users, or, more accurately, maybe it was relevant, but performance was *not* a salient feature of Ada/Ed. Other vendors may simply not be interested in customers who require ACES testing, and that's quite a reasonable position (they won't be able to sell to these customers). You can't even force all vendors to validate, let alone run other tests in any compleely general way. You *can* and *should* request vendros to meet your needs, and requesting ACEC test results is certainly reasonable (it is a bit of a surprise to me that the F22 program did not require ACEC testing). As for publishing test results, as far as I know ACEC test results can be published -- this is not true of all test suites, there are often restrictions. Spec results can only be published in a certain manner, and as far as I know results from the NPL stress tests cannot be published at all (although this might have changed) -- basically the idea behind such restrictions is to avoid over-focus on the results of a particular suite, or to avoid its use in advertising. That decision is of course up to the vendor of the test suite. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: The Ada Compiler Evaluation System 1996-04-17 0:00 The Ada Compiler Evaluation System Philip Brashear 1996-04-18 0:00 ` Ken Garlington @ 1996-04-29 0:00 ` Laurent Guerby 1 sibling, 0 replies; 8+ messages in thread From: Laurent Guerby @ 1996-04-29 0:00 UTC (permalink / raw) Robert Dewar writes [deleted] : P.S. on another matter, I think the important distinction that Gary was ^^^^ Hum, I don't think Gary is posting here ;-). : trying to make was that you can pass or fail the ACVC tests, they are : obvjective in that sense. You cannot pass or fail the ACEC tests -- any ^^^^ : more than you can pass or fail the SpecMark (which incidentally is : a TERRIBLY misleading indication of microprocessor performance, as : everyone in the industry knows, but everyone keeps advertising it, : because users do indeed depend on it -- this is one of the well known : problems with standard performance tests). [deleted] I think the new name is ACES (fusion ?). Another thing about evaluation (performance) is the configuration problem. It's hard to guess from synthetic benchmarks what will be the performance of your application on your site with your configuration. The only way to have a reliable answer is to test it yourself on your site (of course there's a price for it). -- -- Laurent Guerby, student at Telecom Bretagne (France), Team Ada. -- "Use the Source, Luke. The Source will be with you, always (GPL)." -- http://www-eleves.enst-bretagne.fr/~guerby/ (GATO Project). -- Try GNAT, the GNU Ada 95 compiler (ftp://cs.nyu.edu/pub/gnat). ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~1996-04-29 0:00 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 1996-04-17 0:00 The Ada Compiler Evaluation System Philip Brashear 1996-04-18 0:00 ` Ken Garlington 1996-04-21 0:00 ` Robert Dewar 1996-04-22 0:00 ` Ken Garlington 1996-04-24 0:00 ` Robert Dewar 1996-04-26 0:00 ` Ken Garlington 1996-04-27 0:00 ` Robert Dewar 1996-04-29 0:00 ` Laurent Guerby
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox