From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,42427d0d1bf647b1 X-Google-Attributes: gid103376,public From: Ken Garlington Subject: Re: Ada Core Technologies and Ada95 Standards Date: 1996/04/03 Message-ID: <3162B080.490F@lfwc.lockheed.com> X-Deja-AN: 145678719 references: <00001a73+00002c20@msn.com> content-type: text/plain; charset=us-ascii organization: Lockheed Martin Tactical Aircraft Systems mime-version: 1.0 newsgroups: comp.lang.ada x-mailer: Mozilla 2.01 (Macintosh; I; 68K) Date: 1996-04-03T00:00:00+00:00 List-Id: Robert I. Eachus wrote: > > The naming of ACVC 1.x tests was such that they could be directly > traced to the requirements. I don't remember a reverse mapping ever > being published... In my domain, a reverse mapping is always done by the test developers, to demonstrate effective test coverage. Anyway, since ACVC isn't intended to measure product quality, I don't know it matters... > > Since I expect to be one of the end users, my being unfamiliar > > with the process would tend to support my statement, wouldn't it? > > You are planning to validate a compiler? Well, possibly, but by "end user" I meant a user of an Ada compiler. See Mr. McCabe's post on what "end user" means for us. For example, even though they aren't conducting the software formal qualification test, we have pilots and flight-line maintainers involved in the development of our tests. > We also have several collections of domain specific tests we often > run. If your project is going to make heavy use of generics or > tasking, or you are concerned with how the run-time interacts with > Unix signals, much better to find out up front if the compiler suits > your needs on your project up front. A centralized validation > facility can never do that for you, they just set a reasonable base > level. Unfortunately, I can never tell the criteria by which "reasonable" coverage was established. However, as I noted above, I'll take it on faith that the current ACVC is as good as it can be. > Very true, but some regression tests have made it into the ACVC > suite because they were indicative of more general problems. In any > case, every decent compiler vendor maintains a suite of regression > tests, and runs it regularly. Sounds like something I would find in a set of guidelines for a "good" development process for an Ada compiler. Does such a set of guidelines exist? Is the existence of a suite of regression tests sufficient to determine if a compiler vendor (and by extension a compiler product) is "decent"? Is it necessary, for that matter? > The only cost--if you never find any > bugs--is the disk space and the weekend computer cycles. But you do > find bugs, and often for the reason that gives the suite its name. Why would the cost of regression testing be so cheap, and the cost of other kinds of testing be so dear that they are unmanageable for a system of the complexity of an Ada compiler? > > Also, if the regression suite is that big, is that a good or a bad > > thing? > > Neither, it is a symptom of a mature compiler. This sounds like you are saying, "the bigger the regression test suite, the more mature the compiler." Does this mean that if compiler A has 10,000 tests, and compiler B has 100,000 tests, that compiler B is more mature? > > Again, a difference in philosophy: In my domain, _not_ having a > > regression test for every bug found is out of the question. > > The AVO tries to put in regression tests for every bug in the ACVC > suite. How many attempts were made in the last three years to add a regression test to the ACVC? How does that compare to the list of known bugs in Ada compilers? I guess I'm still operating from ignorance: Dr. Dewar seemed to think that it wasn't possible to try to put in regression tests for every bug, but you're saying this is what is attempted? Perhaps we're talking about different types of bugs? > > However, if I go to my customer, and say: "Our systems really > > complex, and there's no way to develop a test suite that > > guarantees bug-free operation, so you'll just have to live with > > the current defect rate," he'll nod knowingly through the first > > two statements, and cheerfully chop my head off after the > > conclusion. That's the environment that I (and Mr McCabe, I > > suspect) live in. > > How about you change the conclusion to "We can keep reducing the > defect rate, but we can never get it to zero." You might get to keep > your head. Do we believe that there is something in the works within the Ada community to keep reducing the defect rate? Where is this effort documented? If there is such a plan, discussing it would be a welcome change from the "can't be done" answer I've been given so far in this thread! > > As a result, I have to define a means to reduce that error rate > > over time. I have to measure that error rate, to show that it is > > being reduced. And, (worst of all!) I have to share that error > > rate with my customer. When the measured rate fails to meet my > > goals, I get clobbered. When it meets or exceeds my goals, do I > > get flowers? No! But, at least I don't get clobbered. > > Exactly. Exactly what? Are you saying that this is the process commonly used by Ada tool vendors? > > I understand that commercial software can and does work > > differently. I also know that talking about a set of different, > > competing companies as though they were a single entity ("the Ada > > vendors") is also naive. > > Hmmm... SOME of the commercial software markets work much > differently. Others, especially in where systems are safety critical > look very much the same. Do you believe the Ada compiler vendor market looks more like "some" markets, or like a "safety critical" market? I think that question is pretty much at the heart of my grousing, and I suspect at the heart of Mr. McCabe's statements as well. > > -- > > Robert I. Eachus > > with Standard_Disclaimer; > use Standard_Disclaimer; > function Message (Text: in Clever_Ideas) return Better_Ideas is...