From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,e4e62e0a73fb6667 X-Google-Attributes: gid103376,public From: Ken Garlington Subject: Re: The Ada Compiler Evaluation System Date: 1996/04/22 Message-ID: <317B7757.4849@lmtas.lmco.com> X-Deja-AN: 150834460 references: <4l2nt1$p4k@ns1.sw-eng.falls-church.va.us> <31761BD5.7D11@lmtas.lmco.com> content-type: text/plain; charset=us-ascii organization: Lockheed Martin Tactical Aircraft Systems mime-version: 1.0 newsgroups: comp.lang.ada x-mailer: Mozilla 2.01 (Macintosh; I; 68K) Date: 1996-04-22T00:00:00+00:00 List-Id: Robert Dewar wrote: > > I think you answered your own question. The point is really quite simple. > (Again I remind you we are talking here only about internal DoD policy). > The DoD should require of all Ada vendors only those quality criteria > which arguably apply in a cost-effective manner across the board to all > DoD users of Ada. OK. Given that" [t]he DoD should require of all Ada vendors only those quality criteria which arguably apply in a cost-effective manner across the board to all DoD users of Ada," what should be included in that criteria? Should it be the ACVC? Should it be a different ACVC? Should it be something in addition to the ACVC? Should it be something instead of the ACVC? (Deja vu!) > Why should we do even that much? That's actually not entirely clear. I agree. Is there anyone that can clarify this? > If > all procurement officers in the DoD were to require all vendors to do > XXX for their project in order to sell their compiler, that would in > practice have the same effect as a mandate, i.e. vendors who want to > sell to the DoD would have to do XXX. Of course, as is the case even > with the "requirement" of validation, if you don't want to sell to the > DoD, you can ignore even validation. And if all users required all vendors to do XXX for their project, regardless of what the procurement officers requested, then that also would be the same as a mandate. It would be stronger than a DoD mandate, in fact, since it would include commercial users. Are there measures of compiler quality that all users should require? > I guess the argument in favor of doing that much is not so much to control > vendors as to control procurement officers. If the DoD has determined that > for it purposes all Ada compilers should satisfy XXX, then presumably it > has an interest in mandating XXX and taking away the freedom of procurement > officers to say "I don't care about XXX, I would rather save money." That > seems to be the motivation behind requiring validation. That's an interesting view of the situation, and you may well be right that the DoD requires a common measure to control procurements. However, it does beg the question, "Why ACVC as the best/only measure?" > The requirement for > validaton does of course increase compiler costs (e.g. in comparison with > C++) but if the DoD determines that this cost increase is commensurate > to the quality improvement, then the general requirement makes sense I absolute agree with this statement. Can we now talk about the "if" embedded within it? Does the DoD have information that validates the ACVC's role in quality improvement? Does anyone outside DoD have this information? > in practice I think this inrease > in cost is fairly minimal, yes validation is expensive, but the > expense is not huge in proportion to other factors -- talking from > one vendors point of view anyway). OK, but to compute return on investment, we need more than the cost. We need the benefit. The cost may not be huge, but a more interesting question is, "Is it justified?" Alternately, there may be measures that are more expensive, but in fact bring much more quality. In the case of SEI or ISO 9001 measures, there might be even more to the story. There might be a higher up-front cost, but a lower cost (compared to ACVC) over the longer term. > Going back to the original issue, Why not do more? i.e. why not have DoD > universally require more than ACVC testing. There are two subquestions: > > 1) would requiring more increase quality for all possible applications > of Ada in the DoD? > > 2) even if the answer to 1) is no, should it be done anyway? > > Let's take 1) first. If the answer is yes, the DoD still needs to do > an assessment of whether it would increase the cost unacceptably. This > is certainly difficult to do assess across the board. The current statement > of policy is a judgment that either the answer to 1) is no, or that if > it is yes, then the cost would be too high. I think the current statement of policy might _imply_ such a judgement, but I don't see that it states such a judgement. There is an alternaive explanation: Maybe no one in DoD has asked these questions, or had the time to answer them. Maybe they require NIST certification because they always require certification. > The answer to 2) is almost certainly no. The issue is that if you mandate > extra testing or certification that does NOT help quality for certain > applications, it will most certainly DECREASE quality for these applications > by diverting resources. Granted. However, the converse -- extra testing that increases quality sufficiently for a good ROI, or just a better ROI than ACVC -- would in fact improve resource allocation, right? > It's quite interesting that no one else has hopped into this thread. > Certainly I assume that all the vendors and users reading this thread > are interested in improving quality. I agree. I really expected a broad sampling of opinions on this topic. > Q. Are you interested in improved quality > > A. Of course. > > Q. Do you think all compiler vendors should be required by DoD to do XXX? > > A. Hmmm! not so simple to answer, we would have to look at XXX very closely. Absolutely - it's a difficult question to answer. That doesn't mean it shouldn't be asked, however. > Ken worries that if every user does their own evaluation that is inefficient. > Sure that is true, but no centralized DoD testing is going to relieve you > of that requirement. I also agree with this. However, it might allow the users to focus more on domain-specific tests, and less on tests that could be done globally. Right now, we do a lot of doamin-specific testing, which means we don't have the resources to do a lot of generic testing. I would argue both kinds of testing are important. > Early on some naive people thought that validation > was some marvellous panacea that would relieve them of the need to > carefully evaluate competing compiler products. What Ken seems to be > looking for at times is some other formula that would provide that > panacea, and I am afraid it is not possible. Not at all. I just want to maximize the ROI for mandated, global testing. It may not be possible to have an effective general-purpose measure of compiler quality beyond the ACVC. It may also be true that the ACVC provides an excellent ROI. However, I think there is a lot of evidence in the commericial software world that indicates effective global measures are possible. > Even if all sorts of > extra requirements were placed, it would not relieve someone starting > out on a major project from the need to evaluate to see if the tools > to be used met the needs. No argument there. However, it would be nice to pick up a report containing a list of common compiler measures, rather than doing those measures yourself, and then using that as a basis _in combination with_ your domain-specific tests. This certainly happens in the hardware world, and in some software domains today. > On the other hand, you certainly don't have to do stuff yourself. You > can make the vendor do it. No one suggests that every user do their > own ACES testing. > Let's suppose that you have determined that ACES testing is useful. In > that case you should simply only look at products for which ACES testing > results are available (this has certainly been done in some cases). Are ACES results kept in the public domain? Has ACES been performed on most Ada compilers? Can the results be interpreted in a way that has value for most Ada users? If the answer to all three is yes, then I would say ACES is an example of the very thing I'm looking for. > Similarly, if you think ISO 9000 certification is useful, then by all > means require it (other users of Ada have done so, Alsys did not get > ISO 9000 certification for its own internal requrements, but because > customers required it). OK, I will! :) However, there's a couple of problems: First, If I set a bar on compiler quality at the level of ISO 9001, not all vendors are interested in reaching for that bar regardless of how much I'm willing to invest (as far as I can tell). This drastically limits my choices, in a market where I don't have a large number of choices to begin with (depending upon the host/target sets I need to support). This adds risk to my project. The second problem is that the Ada market is much smaller than the market for languages such as C or C++. A large player in the market, if he requests a measure that is not value added, can really do serious damage to a vendor. I know. I have made this mistake. However, some vendors seem reluctant to discuss whether or not a user requirement makes sense, particularly when there's a lot of money riding on the sale. So, from my perspective, it makes more sense to ask these questions outside of a procurement, and try to get honest feedback from relatively disinterested parties. That way, when I go into a procurement and require ISO 9001, SEI III, or whatever, I have confidence that this makes sense to the vendor commercially and has a good ROI with respect to compiler quality. > Ken, in the case of your project, were extra requirements like this > set down? If not why not? Yes, we have a spec for every project. Some of these extra requirements are now sold commercially. One of these requirements was for my vendor to implement the equivalent of pragma Reviewable. Now, this is a standard part of the Ada 95 language, since enough users apparently thought this function was needed to include it in the standard (although part of an appendix). I'm looking for the same concept, but for that which can't be put in the Ada standard: measures of compiler quality Let me digress briefly into how my group buys compilers: 1. We write a spec that describes what we want. It talks about features we need, capacity limits, performance, etc. It hasn't historically talked much about robustness, etc. since we haven't in the past had ways to measure such things. 2. We create benchmarks. Since coding hasn't started yet, we use samples of code from the previous project, which we try to get from each domain that will be using the compiler. 3. We release RFPs that include the spec, the SOW, and all the good contractual language. 4. We evaluate the potential vendors, by checking their proposals against the spec, running the benchmarks, and so forth. 5. We select a compiler, get it in-house, integrate it with the rest of the software engineering environment, and so on. The problem is threefold: 1. The specs and benchmarks have to be written very early in the program, before the users know enough about their requirements to say for certainty what they think they need. (In addition, they are busy doing trade studies and so forth, and so don't have time to spend thinking about coding!) 2. Our requirements are always based on the last program. The next program is always different. 3. Until recently, there weren't any commonly used measures of software quality, so our specs never addressed them. As a result, it occurs to me that it would be far smarter to de-emphasize the specs and benchmarks (but not eliminate them), and to emphasize general measures of software quality. Presumably, if a vendor has a high-quality product and process, then (a) the users will be happier and (v) the vendor will be more likely to cope with the inevitable changes the users will require, either those that are defined in the spec or those that come up later. It also occurs to me that, since these measures will be general (not tied to a specific domain), and if users other than myself wanted them, I might not have to bear the full cost for them. Even better, what if some other user or set of users (including the DoD) has already paid for those measures, or could be convinced to pay for those measures, for a large set of the available compilers? Then I would really be smart to add those measures to my spec, since I could get them for little or no cost! Of course, to know which measures make sense, I would have to ask compiler vendors their opinion. If only one vendor thought a particular measure made sense, then I might be locking myself into that one vendor, which is a risk. On the other hand, if most vendors seemed to think ISO 9001 (for example) was reasonable, and the only reason they hadn't done it was a lack of funds, then at least I had somewhere to start. > Ken, what's so terrible about letting the market place decide what > requirements to place on tools that are acquired, instead of concentrating > on having DoD make general requirements that might or might not be suitable > for your particular project. I don't limit this discussion to what DoD should or shouldn't do. I am perfectly willing to make this a market-driven, unofficial standard. So, what should users be asking for from their vendors, that would be a common measure? > Yes I know, you keep asking why, if this argument is valid, should it > not be applied to the ACVC itself? The answer is that there is a general > consensus in the DoD that this is a good idea. Furthermore, it is not > unusual historically. There was also a general consensus in the DoD on using MIL-STDs instead of equilalent commercial standards, and it wasn't unusual historically, either. I suspect the trend toward acquisition reform started with one person asking the dreaded question, "Why do we do it this way?" The first time the question was asked, they probably got the same answer you just gave. Frankly, it's a crappy answer from a quality perspective! > Ken wants the mandate strengthened in this area, but it is not clear > there is any consensus for such strengthening. It's not clear to me, either. That's why I'm asking these questions. How else would we discover if there was such a consensus? > P.S. Ken, you recommend improving the quality of the ACVC suite. Fine, but > can we have some specific constructive suggestions? Sure. Change the scope of ACVC as follows: 1. Require vendors to attach their software development plans to their ACVC test reports, along with a signed statement that the product was developed using the plan. 2. Require vendors to attach their internal software test procedures to their ACVC test reports, along with a signed statement that the product passed those tests. 3. Require vendors to provide updates to the ACVC based on their regression test suites, where those tests (a) can be made public and (b) can be written as a general-pupose test. 4. Require vendors to sign a statement that they conform to ISO 9001 and/or SEI III. -- LMTAS - "Our Brand Means Quality"