Re: Ada Core Technologies and Ada95 Standards

[Ken Garlington <garlingtonke@lfwc.lockheed.com>]

Robert Dewar wrote:
> 
> Ken said
> 
> "I know that NPL has a tool that they sell that tests Ada compilers for bugs, tha
> t
> apparently provides much more coverage than the ACVC. Why should such a tool
> exist outside of the validation/certification process?"
> 
> This is not true at all, and I guess Ken is only aware of this tool by
> rumour, since if he had used it he would now that it is not ni the business
> AT ALL of providing coverage testing.

Slow down with the speed reading, Dr. Dewar. I said "provides more coverage,"
not "does coverage testing." There is a difference!

The rumor, which some guy named Dr. Brian Wichmann fed me, is that this tool
does the following:

> Instead this is a stress testing tool, it generates random very complex
> (and generally very unrealistic) examples of expressions and other
> constructs to see if the compiler can be broken by such stress testing.

I won't comment on the "unrealistic" part, but at least Dr. Wichmann seems to
think that this tool tells me more about compiler quality than if I don't use it.

> The tool is incidentally available from NPL to users, so Ken it is
> certainly something you could use to test a compiler yourself.

I could also run the ACVC myself. What's the point? 

DO THE MATH. Each user pays for [fill in your favorite process/tool/test here],
or it's done once and the results made available to all users. I wonder which
is more cost-effective?

> In fact, I talked to Brian Wichman (the author of this tool and a similar
> one for Pascal), and the results they have obtained with these tools are
> quite surprising at least to me, in the extent to which they show quality
> differences between Ada and Pascal compilers. Most (all?) of the Pascal
> compilers they have tested have exhibited safety defects (defined as the
> generation of incorrect code). None of the Ada compilers have shown
> safety defects -- they have managed to break them but not persauded
> then to generate wrong code.

"They have managed to break them."

Sounds like a good test to me. Sounds like a test that found something that
neither the vendor's process, nor the ACVC, found before release. Too bad compiler
vendors don't use tools like this _before_ their products are released, to improve
the quality of their software.

Of course, no vendor could afford such a tool, nor could the AVOs afford to
have such a tool. Out of the question! So, I guess the end users will have to
continue to be the guinea pigs, and test the compilers.

By the way, these results surprise me too, since I have certainly managed
to get Ada compilers to generate wrong code. Of course, that doesn't make this
test bad, since no test can guarantee that all possible errors can be caught.
(Notice: _I_ said this.) As you point out:

> Now in practice, I would expect that big projects such as Ken's can point
> to safety defects (defined this way) in the compilers they have used, and
> just as the ACVC cannot 100% guarantee conformance, the NPL tests cannot
> 100% guarantee safety, but they are a measure.

But not a necessary measure, since the vendor process + the ACVC will already
provide high-quality compilers. The tool is unncessary, if I understand your
previous posts. Right?

> I find it interesting that
> the Ada compilers fair so much better than the Pascal compilers. Brian at
> least ascribes this at least in part to the ACVC process.

Well, as long as Ada compilers are higher quality than Pascal compilers, I guess
I have no reason to gripe, eh?

Tom Peters tells this wonderful story about an unnamed company, who hired Mr. Peters
as a quality consultant. At a meeting, one of the managers in frustration said, "Hey!
Get off our backs! We're no worse than anyone else!"

Mr. Peters liked that last expression so much, he put it on a business card. I guess, in
the Ada world, it would have looked like:

      Joe Smith
      XYZ Ada Tools
      "We're No Worse Than Anyone Else, and Better Than Pascal!"