From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,42427d0d1bf647b1 X-Google-Attributes: gid103376,public From: eachus@spectre.mitre.org (Robert I. Eachus) Subject: Re: Ada Core Technologies and Ada95 Standards Date: 1996/04/03 Message-ID: X-Deja-AN: 145616459 references: <00001a73+00002c20@msn.com> organization: The Mitre Corp., Bedford, MA. newsgroups: comp.lang.ada Date: 1996-04-03T00:00:00+00:00 List-Id: In article <3160EFBF.BF9@lfwc.lockheed.com> Ken Garlington writes: > In the ACVC 1.0 sources I received, each test had an identifier. I > did not receive a cross-reference of that identifier to the > requirements, as I recall. The naming of ACVC 1.x tests was such that they could be directly traced to the requirements. I don't remember a reverse mapping ever being published, but I remember spending a couple hours with the Implementors Guide, a list of the tests in ACVC 1.4 and Multics Emacs to create one. For DPS-6 Ada, we used it to trace failed tests to requirements--it was often easier than figuring out what the test was supposed to be testing--and for both DPS-6 Ada and Ada/SIL we kept a copy with annotations as to non-applicable tests or edited and why the tests were ignored or modified. (I have to admit that a LOT of the comments in the documents were of the form "STUPID TEST: ..." For example one test had two programs, one of which opened a temporary file and attempted to write it's name to another file. The other program tried to open the temporary file, using that name, and read the contents. It took a lot of back and forth to agree that the "right" behavior on our system was to raise USE_ERROR for NAME of a temporary file, but most of that was over NAME_ERROR vs USE_ERROR. Of course the test did not anticipate the possibility that the call to NAME would fail.) > Since I expect to be one of the end users, my being unfamiliar > with the process would tend to support my statement, wouldn't it? You are planning to validate a compiler? (This is not a sarcastic question. We often put compilers through the ACVC process here because there are potentially significant differences between the system as validated and the configuration being used on a project. Such an informal validation takes a few man-days, including comparisons to the original VSR.) > 1. Then, perhaps, Mr. McCabe and I want something other than a > general validation facility. How about standards that each vendor > must meet for development and test -- SEI, ISO, and/or something > else? We also have several collections of domain specific tests we often run. If your project is going to make heavy use of generics or tasking, or you are concerned with how the run-time interacts with Unix signals, much better to find out up front if the compiler suits your needs on your project up front. A centralized validation facility can never do that for you, they just set a reasonable base level. Robert Dewar said: > > It would not be practical to incorporate all test progrms for all bugs > > found in all compilers into the ACVC (it would rapidly have tens of > > thousands of tests, and become completely unmanagable). Very true, but some regression tests have made it into the ACVC suite because they were indicative of more general problems. In any case, every decent compiler vendor maintains a suite of regression tests, and runs it regularly. The only cost--if you never find any bugs--is the disk space and the weekend computer cycles. But you do find bugs, and often for the reason that gives the suite its name. (Usually this happens when a module is rewritten from scratch because it contains too many bug fixes. Some of the fixes get lost in the rewrite. In some cases this happens becuase the original fix got buried under other fixes--remember why you rewrote this.) > Also, if the regression suite is that big, is that a good or a bad > thing? Neither, it is a symptom of a mature compiler. Some of the tests come from actual bug fixes, many more are unit tests written to verify additional features at each build. Again, there is no reason not to run a test again and again, once you have gone to the trouble of writing (and verifying) it. > > Also, the effort of taking every > > bug and putting it into ACVC form is out of the question. > Again, a difference in philosophy: In my domain, _not_ having a > regression test for every bug found is out of the question. The AVO tries to put in regression tests for every bug in the ACVC suite. (And in some cases the ARG slaps them down. If an ACVC test tests for something harmful, then no test is the right position. The ARG created the category Pathology just for cases like testing 'TERMINATED for a task outside the scope of its master.) But the individual compiler vendors maintain their own regression suites. > However, if I go to my customer, and say: "Our systems really > complex, and there's no way to develop a test suite that > guarantees bug-free operation, so you'll just have to live with > the current defect rate," he'll nod knowingly through the first > two statements, and cheerfully chop my head off after the > conclusion. That's the environment that I (and Mr McCabe, I > suspect) live in. How about you change the conclusion to "We can keep reducing the defect rate, but we can never get it to zero." You might get to keep your head. > As a result, I have to define a means to reduce that error rate > over time. I have to measure that error rate, to show that it is > being reduced. And, (worst of all!) I have to share that error > rate with my customer. When the measured rate fails to meet my > goals, I get clobbered. When it meets or exceeds my goals, do I > get flowers? No! But, at least I don't get clobbered. Exactly. > I understand that commercial software can and does work > differently. I also know that talking about a set of different, > competing companies as though they were a single entity ("the Ada > vendors") is also naive. Hmmm... SOME of the commercial software markets work much differently. Others, especially in where systems are safety critical look very much the same. -- Robert I. Eachus with Standard_Disclaimer; use Standard_Disclaimer; function Message (Text: in Clever_Ideas) return Better_Ideas is...