Re: Ada Core Technologies and Ada95 Standards

[Ken Garlington <garlingtonke@lfwc.lockheed.com>]

Robert Dewar wrote:
> 
> Ken Garlington asks why it is infeasible for a compiler vendor to deliver
> the source code to the AVF for anaysis.

Actually, I didn't ask this, but we can talk about it if you like...

What I actually asked was, "Is there some way to modify the scope of the ACVC
process to improve compiler quality across all vendors? Or, is there something
outside the scope of the ACVC that could be done to improve compiler quality
across all vendors?"

Your answer: No, because we'd have to make an _investment_ to improve
compiler quality. To get to 100% quality (whatever that means) would take
too much money (and is technically infeasible). Therefore, no investment
should be made.

> Ken, you have some experience here. What would you say is the cost of
> analysis and thorough testing of half a million lines of someone elses
> code, under the conditions that the code is, throughout, extremely
> complex.

The cost of IIV&V? I don't know the exact F-22 figure at the moment, but
it's probably significantly less than 5% of the development cost. IIV&V
is done on far more than a mere 500K SLOCs on F-22. I recommend AFSC Pamplet
800-5, which helps estimate such costs, and also explains IIV&V. Based on your
discussion below, I'm guessing you're not familiar with this process.

> Remember that a typical compiler has had several hundred
> person years invested in the code, at least this figure is right
> for several Ada compilers that I know of. How much more investment
> would be necessary from the AVF to significantly improve the level
> of confidence on the basis of examination of the source code.

"What statistics there are show that path testing catches approximately
half of all bugs caught during unit testing or approximately 35% of all
bugs.... When path testing is combined with other methods, such as limit
checks on loops, the percentage of bugs caught rises to 50% to 60% in
unit testing."

    - Beizer, Software Testing Techniques, 1983.

So, if path testing were required for Ada vendors - either at the vendor's
site, or at an AVF - this would be the expected benefit over not doing it.

How much would it cost to do path testing for an Ada compiler? I don't know.
Let's ask someone from Rational. They produce TestMate; surely they use it
in their own development process, or at least would know what it would cost!
I bet the folks who build AdaMat, LDRA Analysis, etc. would be happy to provide 
good information in this area, as well.

I don't like focusing on path testing, since there are certainly many other
analyses that could also be done, but that's one idea. Other ideas might be
to audit processes (or use an SEI III/ISO 9000 audit), or do data flow analysis
(see the TRI-Ada '95 paper from Boeing).

> Let's suppose that for this kind of examination and white box testing,
> a figure of 10 lines/day is reasonable (this is ten lines of source code).
> I suspect this number is high, but I deliberately what to be on the high
> side.
> 
> Then we arrive at a figure of 250 person years to evaluate the code
> of an Ada compiler. OK, so that's about 25 million dollars.
> 
> I *think* it is ok to regard this as infeasible :-)

Interesting. So, once you reach 500K SLOCs, you can no longer perform adequate
testing of software. What a relief! Now, if the F-22 fails to satisfy the
customer, I have an ironclad alibi! :)

Of course, no one would do IIV&V this way, so this is a straw man analysis.
Nonetheless, it is gratifying to hear at least one vendor admit that their
product is inadequately analyzed and tested, in order to save development costs.
Or are you saying that, somehow, you do manage to adequately test your product,
despite the exorbitant cost?

> The real point, which you did not address,is that even if you were to
> supply the check for $25 million, it would not solve the problem of
> timely delivery and verification of improvements etc.

Well, I wasn't asked to address this point, but of course IIV&V only would
have to address the delta changes and their interfaces.

Are you saying that we're wasting money re-running ACVC tests on changed
products? Maybe we could use that money to do process audits! See, that's
exactly the kind of thinking I'm looking for here. Good idea!

> Furthermore, we are still missing a formal specifcation of Ada 95 against
> which to formally measure compliance.

We don't have a formal specification of the F-22 software, either.

Can you come to our first flight readiness review, and explain to the pilots
why we're not able to give him any confidence in the performance of the system
because we're missing a formal specification?

> Ken, in your message, you again refer to users expecting the ACVC suite
> to guarantee conformance to the standard.

I did? Must have been my evil twin.

What I actually asked was, "Is there some way to modify the scope of the ACVC
process to improve compiler quality across all vendors? Or, is there something
outside the scope of the ACVC that could be done to improve compiler quality
across all vendors?"

> How many times does it have
> to be said? The ACVC suite cannot do this, does not attempt to do this,
> and anyone who thinks it does do this, or could do this, is mistaken!

Probably as many times as I (and other users) have had to say:

"Is there some way to modify the scope of the ACVC process to improve compiler
quality across all vendors? Or, is there something outside the scope of the
ACVC that could be done to improve compiler quality across all vendors?"

> Once again, I refer you to John Goodenough's writings on the subject,
> and to the other material I mentioned before.

And once again, how about the actual _name_ on the paper? Where on the
Internet it is located?

(See separate message for a review of the "other material").

> P.S. If you would like to send a check for $25 million to ACT, I think
> I can promise that 5 years from now we wlil have a compiler that is
> much closer to conforming to the standard (of course I can also promise
> this if you *don't* send us the $25 million :-)

Interesting. Your process for improving the quality of your product is
unrelated to the available resources? Wish _we_ had a system like that.
(Or maybe I don't.)

I notice you use the word "conformance" rather than "quality". Are these
synonyms, to you? They aren't to me. I suspect they aren't to Mr. McCabe,
or most other vendors.

Again, I think it's a matter of culture. We're both speaking English (more
or less), but discussing completely different subjects. Since you've already
answered my question, I'm not really sure why you're wasting your valuable
time continuing to discuss it...