Re: Ada Core Technologies and Ada95 Standards

[Ken Garlington <garlingtonke@lfwc.lockheed.com>]

Robert I. Eachus wrote:
> 
>    The naming of ACVC 1.x tests was such that they could be directly
> traced to the requirements.  I don't remember a reverse mapping ever
> being published...

In my domain, a reverse mapping is always done by the test developers,
to demonstrate effective test coverage. Anyway, since ACVC isn't 
intended to measure product quality, I don't know it matters...

>   > Since I expect to be one of the end users, my being unfamiliar
>   > with the process would tend to support my statement, wouldn't it?
> 
>    You are planning to validate a compiler?

Well, possibly, but by "end user" I meant a user of an Ada compiler. See
Mr. McCabe's post on what "end user" means for us. For example, even though
they aren't conducting the software formal qualification test, we have pilots
and flight-line maintainers involved in the development of our tests.

>    We also have several collections of domain specific tests we often
> run.   If your project is going to make heavy use of generics or
> tasking, or you are concerned with how the run-time interacts with
> Unix signals, much better to find out up front if the compiler suits
> your needs on your project up front.  A centralized validation
> facility can never do that for you, they just set a reasonable base
> level.

Unfortunately, I can never tell the criteria by which "reasonable"
coverage was established. However, as I noted above, I'll take it on
faith that the current ACVC is as good as it can be.

>   Very true, but some regression tests have made it into the ACVC
> suite because they were indicative of more general problems.  In any
> case, every decent compiler vendor maintains a suite of regression
> tests, and runs it regularly.

Sounds like something I would find in a set of guidelines for a "good"
development process for an Ada compiler. Does such a set of guidelines
exist? Is the existence of a suite of regression tests sufficient to
determine if a compiler vendor (and by extension a compiler product)
is "decent"? Is it necessary, for that matter?

> The only cost--if you never find any
> bugs--is the disk space and the weekend computer cycles.  But you do
> find bugs, and often for the reason that gives the suite its name.

Why would the cost of regression testing be so cheap, and the cost of
other kinds of testing be so dear that they are unmanageable for
a system of the complexity of an Ada compiler?

>   > Also, if the regression suite is that big, is that a good or a bad
>   > thing?
> 
>    Neither, it is a symptom of a mature compiler.

This sounds like you are saying, "the bigger the regression test suite,
the more mature the compiler." Does this mean that if compiler A has
10,000 tests, and compiler B has 100,000 tests, that compiler B is more
mature?

>    > Again, a difference in philosophy: In my domain, _not_ having a
>    > regression test for every bug found is out of the question.
> 
>    The AVO tries to put in regression tests for every bug in the ACVC
> suite.

How many attempts were made in the last three years to add a regression
test to the ACVC? How does that compare to the list of known bugs in Ada
compilers? I guess I'm still operating from ignorance: Dr. Dewar seemed
to think that it wasn't possible to try to put in regression tests for
every bug, but you're saying this is what is attempted? Perhaps we're
talking about different types of bugs?

>   > However, if I go to my customer, and say: "Our systems really
>   > complex, and there's no way to develop a test suite that
>   > guarantees bug-free operation, so you'll just have to live with
>   > the current defect rate," he'll nod knowingly through the first
>   > two statements, and cheerfully chop my head off after the
>   > conclusion. That's the environment that I (and Mr McCabe, I
>   > suspect) live in.
> 
>   How about you change the conclusion to "We can keep reducing the
> defect rate, but we can never get it to zero."  You might get to keep
> your head.

Do we believe that there is something in the works within the Ada
community to keep reducing the defect rate? Where is this effort documented?
If there is such a plan, discussing it would be a welcome change from
the "can't be done" answer I've been given so far in this thread!

>   > As a result, I have to define a means to reduce that error rate
>   > over time. I have to measure that error rate, to show that it is
>   > being reduced. And, (worst of all!) I have to share that error
>   > rate with my customer. When the measured rate fails to meet my
>   > goals, I get clobbered.  When it meets or exceeds my goals, do I
>   > get flowers? No! But, at least I don't get clobbered.
> 
>    Exactly.

Exactly what? Are you saying that this is the process commonly used by
Ada tool vendors?

>   > I understand that commercial software can and does work
>   > differently. I also know that talking about a set of different,
>   > competing companies as though they were a single entity ("the Ada
>   > vendors") is also naive.
> 
>    Hmmm... SOME of the commercial software markets work much
> differently. Others, especially in where systems are safety critical
> look very much the same.

Do you believe the Ada compiler vendor market looks more like "some"
markets, or like a "safety critical" market? I think that question is
pretty much at the heart of my grousing, and I suspect at the heart of
Mr. McCabe's statements as well.

> 
> --
> 
>                                         Robert I. Eachus
> 
> with Standard_Disclaimer;
> use  Standard_Disclaimer;
> function Message (Text: in Clever_Ideas) return Better_Ideas is...