Re: Tiring Arguments Around (not about) Two Questions

[Ken Garlington <garlingtonke@lfwc.lockheed.com>]

Robert Dewar wrote:
> 
> Ken said (double >> are mine)
> 
> >> Ken Garlington asks why it is infeasible for a compiler vendor to deliver
> >> the source code to the AVF for anaysis.
> 
> >Actually, I didn't ask this, but we can talk about it if you like...
> 
>  There must be two Ken Garlington's around, the other one said in
>  a previous post:
> 
>    "I could think of several ways to include other types of testing in an
>    ACVC, e.g.  a requirement to deliver of source code and supporting
>    data to an AVF for analysis."
> 
>    (this is an exactly cutted-and-pasted quote :-)

It's a quote. However, it isn't a question, so I think you've proven that
I did not, in fact, ask this. Note also that I said "e.g." That's
my own personal shorthand for the phrase "for example." 

> >What I actually asked was, "Is there some way to modify the scope of the ACVC
> >process to improve compiler quality across all vendors? Or, is there something
> >outside the scope of the ACVC that could be done to improve compiler quality
> >across all vendors?"
> 
> >Your answer: No, because we'd have to make an _investment_ to improve
> >compiler quality. To get to 100% quality (whatever that means) would take
> >too much money (and is technically infeasible). Therefore, no investment
> >should be made.
> 
>    You miss the point entirely (seems to be happening consistently, so I
>    must not be clear, hence my attempt to clarify!)
> 
>    Of COURSE! more investment is desirable to improve quality.

Any ideas on where this investment should be applied, to improve quality
"across all vendors"? (Really, you have to stop quoting me without
responding to the quote!)

>    ACT is
>    a very focused company, our only business is improving the quality
>    of GNAT!

Great! Let me know if/when you're interested/have time/have materials
available to actually _discuss_ what you do/plan to do (other than
add more tests to the regression test suite.)

Until then, this statement is pretty much indistinguishable from magic.

>    The issue is whether spending more resources on the ACVC
>    testing is the way to do it. I very much doubt it.

Why? You've agreed that SEI III/ISO 9001 is a good thing, for example.
Why is adding this to the things which AVOs do/collect so terrible?

>    One thing we have not discussed is the ways in which the ACVC can,
>    if not carefully handled, actually *decrease* compiler quality.
>    Right now the rules in the Ada game are that you MUST pass the
>    ACVC tests, above anything else, and in particular, above any other
>    testing or quality control procedures that may make sense.

Right. Is this good or bad, to put the ACVC above other testing or quality
control procedure that may make sense? Possibly, the ACVC should be
changed to focus more on quality control procedures, and less on tests.
Unfortunately, this will never happen. The ACVC is perfect, wouldn't
change a thing!

>    Passing the ACVC suite is not a trivial excercise, and I think all
>    vendors would agree that they have had to devote considerable resources
>    to this task. These are resources not available for other quality
>    improving tasks. This means that we have to be very careful that we
>    do not divert too many resources and reach a point of diminishing
>    returns.

Absolutely! So, let's cut the amount of effort to pass the ACVC 10%, and we'll
get a 10% increase in compiler quality, since the vendors will have more
money for other improvements. Why not?

>    For example: Suppose we spent ten times as much on the ACVC suite, and
>    had ten times the number of tests (that's really the only way I
>    could see spending this investment, since the current tests are
>    as effective as the contractor (SAIC) and the review team know
>    how to make them).

However, those tests only measure conformance, right? Oh, that's right,
conformance = quality. Never mind...

>    As Ken has pointed out there are many other activities besides ACVC
>    testing that can contribute to quality:
> 
>       Process control (with ISO 9000 or SEI audits)
>       Systematic white box testing (e.g. path testing)
>       Stress testing (the NPL tool)
>       Performance testing (e.g. with the ACES and PIWG tools)
>       Systematic regression testing
>       Purpose built test suites for particular compilers
> 
>    All these steps may be extremely useful, and compete with the ACVC
>    process for resources.

Unless, of course, there was a way to _include_ these in the ACVC process.
Nah, impossible! An AVO can ask a vendor, "please sign this statement that
you _didn't_ implement extraneous features," but ask them to sign a statement
that they did "systematic regression testing?" Impossible!

>    If you are acquiring an Ada compiler, you certainly do more in your
>    evaluation than assure that it is validated. You may well, if your
>    experience shows this is worthwhile, require additional testing,
>    or for example, require that the compiler vendor have ISO 9000
>    certification. Certainly IBM did quality audits on Ada vendors
>    internal process control for the NASA space station project, and
>    this seems entirely appropriate to me.

Yep, the users are the guinea pigs, all right.

>    I think part of the confusion here is that Ken (and John) use the
>    ACVC as a kind of code word for "all the things that might be done
>    to ensure quality", but in fact the ACVC is and has always been
>    defined to be restricted to black box conformance testing.

And now, I have to shock you to your very core. I am about to suggest
something so heretical, so impossible, that it will be completely
incomprehensible to you.

Ready?

CHANGE THE DEFINITION. OR, CREATE THE ACQV (ADA COMPILER QUALITY
VALIDATION) PROCESS. I DON'T CARE. EITHER WILL ANSWER:

> >What I actually asked was, "Is there some way to modify the scope of the ACVC
> >process to improve compiler quality across all vendors? Or, is there something
> >outside the scope of the ACVC that could be done to improve compiler quality
> >across all vendors?"

You can quote the questions, but you can't answer them.

>    Perhaps
>    what has gone wrong here is that Ken and other Ada users have been
>    confused into thinking that the ACVC was intended to encapsulate
>    the entire question of quality assessment of compilers, and that
>    was never its intention.

Nope. What has gone wrong is that you can quote the questions, but
can't answer them.

>    Let's go back to the ISO 9000 question for a moment. Suppose that your
>    assessment shows that it is essential that a compiler vendor have ISO
>    9000 certification (some do, Alsys obtained this certification, what
>    was involved was basically a lot of paper work, writing down the
>    procedures that had always been followed in formal and precise form).

By the way, that "paperwork" (having a well-documented process) turns about
to also be part of SEI's process, and in fact there's starting to be a rumor
that you can't be an effective software development organization without
well-documented processes.

>    Then it is entirely reasonable for you to require this as part of
>    your procurement process. Naturally you won't want to do this unless
>    your careful analysis shows that this really does contribute to
>    quality, because otherwise you will be requiring vendors to divert
>    resources, but if your analysis shows it's a good idea, and a good
>    way to spend resources, then fine.

Yep, the users take it on the chin financially again.

>    BUT! It is not appropriate for NIST conformance testing to include this
>    criterion, becaues the Ada standard does not (and could not) say anything
>    about process, since it is a language standard, and hence only defines
>    the semantics of the language. So you cannot look to the ACVC for help
>    here.

Apparently, not without to (oh, my God!) change the scope of the ACVC to
include more than NIST conformance testing. (Still haven't answered the
second question, by the by.)

>    Similarly, you may determine that performance of generated code is
>    critical, and consequently place a lot of weight on the ACES test
>    results (if your analysis shows that the ACES accurately captures
>    qualities that are important to you). But this cannot be part of
>    NIST conformance testing, since the Ada standard does not (and could
>    not) say anything about performance.

Apparently, not without to (oh, my God!) change the scope of the ACVC to
include more than NIST conformance testing. (Still haven't answered the
second question, by the by.)

>    Choosing among available compilers is not an easy task.

Actually, it's very easy. For many host/target pairs, there's only a few vendors.
In some cases, only one. You don't like the quality of that vendor? Tough! Spend
your own money to fix it! Ada users got plenty of cash! They're rolling in it!

>    I think that,
>    particuarly early on, procurement officers hoped that the ACVC let
>    them off the hook -- "I'll stick to validated compilers and I will
>    be guaranteed that I have reasonable tools." Unforunately, it is
>    not so simple, and the evaluation process for tools is much more
>    complex. The ACVC testing helps as a kind of first-cut qualification,
>    but that is all it is, and all it pretends to be. Thorough evaluation
>    will have to go well beyond the "Is_Validated" predicate.

Too bad there's no way to implement a common minimal set of processes/tests/etc.
beyond the ACVC, that could be done once for all users. Oh well, they're just
users. Who cares?

>    That *is* a surprise. I admit I am much more familiar with MoD regulations
>    in the Safety Critical area, and with typical British and European
>    procedures. These tend to be much more oriented to formal specifications
>    and formal proof, and so the certification process tends to be much
>    more of the cost, as far as I can gather.

Oh, you wanted the _certification_ process cost! That's different than the IIV&V
cost, of course. What's certification got to do with my questions?

>   First of all, I know of no data that would suggest that Beizer's results
>   extend to the compiler domain, so that would have to be investigated.

Investigated by whom? There's no incentive to answer my questions, so who cares?

>   The danger here is that you enormously increase the cost of ACVC testing.
>   There would be two ways of implementing what you suggest
> 
>     1. Require witness testing of path testing. This seems entirely
>        infeasible. Right now, witness testing of the absolutely fixed
>        set of tests costs of the order of $20K, and witness testing
>        for full path coverage would require a huge amount of analysis,
>        and a lot of very specialized expertise.

Concur. (Surprise!)

> 
>     2. Require a DOC-like document to be signed saying that full path
>        testing should be done

Say, that's a good idea. Wish I'd thought of it.

>   Either of these approaches would in addition require the technical work
>   for full path testing.

Unless you only required 25%/50%/75% path coverage, of course.

>   One very significant problem would be the issue
>   of deactivated code. ... I suspect that systematic testing of
>   a complete compiler would indicate that this problem is intractable
>   for compilers, or at least very difficult.

Who can say? It's not worth exploring, of course.

>   In any case, the issue is whether you would increase quality or decrease
>   quality by this means. It seems to me that this would put too much
>   emphasis on conformance, and too little emphasis on performance.

Are there tests/process requirements/etc. that would add emphasis to performance?
(I don't know what the distinction is in your mind between conformance and
performance, but maybe I'll get a constructive sugesstion by asking the question.)

>   Note also that in the case of optimization algorithms, proving that they
>   do not violate the language rules is significant, but if that is all you
>   do, you miss the point! You want to concentrate at least some effort on
>   showing that the optimizations work.

Why bother?

>   Looking at our customer base, the problems that people have (that are
>   due to GNAT, rather than customer code) fall into several different
>   categories.... Quality improvement for GNAT involves addressing all of these areas.

No it doesn't! (To which you can reply, "Yes, it does!" etc. etc.)

Let me know if you ever answer my two questions.


>   The
>   ACVC tests concentrate entirely on points 1 and 2.

No, they don't. These tests are not designed to find compiler bugs (safe or
unsafe). They are there to provide some level of assurance as to conformance
to the language specification, which is (as you keep point out) a different
issue.

> Certainly these are
> important areas, especially 2, and the few times we have run into code
> generation problems, we certainly consider them to be of the highest
> priority.

Too bad there's no community-wide initiative to decrease problems in areas
1 and 2 (or 3 or 4 or...)

>   Your suggestion of path testing still adds only to points 1 and 2, and
>   I fear in fact that your entire emphasis is on points 1 and 2.

"Let's go back to the ISO 9000 question for a moment."

How come you can quote my suggestions, but not remember them?

But never mind that. I'll gladly accept your statement that we need to
address 3-8. So, in the context of 1-8, let me pose two questions:

"Is there some way to modify the scope of the ACVC
process to improve compiler quality across all vendors? Or, is there something
outside the scope of the ACVC that could be done to improve compiler quality
across all vendors?"

Can you answer these for 1-8 any better than for 1-2?

>   The
>   trouble is that for most of our customers, 3-8 are also important.

Certainly, it's trouble for the users!

>   Again, the issue is that the domains are very different. Your assumption
>   that F22 techniques and observations can apply to compilers are no more
>   valid than if I thought that you should easily be able to make a
>   retargetable flight software program that would work on the F22 or 777
>   with minimal work :-)

Actually, the same "techniques and observations," surprisingly enough, can
apply to both the F-22 and 777.

Of course, I'll gladly accept that your domain requires different processes,
which is why I couched specific proposals as suggestions. Care to fill the
void with suggestions that _will_ work in your domain? The only one I've
heard is "let the user do it." You are right, in that this suggestion won't
fly (so to speak) in MY domain.

> >Are you saying that we're wasting money re-running ACVC tests on changed
> >products? Maybe we could use that money to do process audits! See, that's
> >exactly the kind of thinking I'm looking for here. Good idea!
> 
>   Sorry, I never said that, I think it is definitely very useful to rerun
>   the ACVC suite whenever any change is made to the compiler. We depend
>   on this as one (among several) of our continuing internal quality audit.

Care to name something new in this "several"?

>   I do find that surprising. Talking to the folks doing similar development
>   in England, they seem to be very much more focused on formal specifications.
>   Of course ultimately one important thing to remember is that the requirement
>   on your F22 software is not that it be 100% correct, but that in practice
>   it be 100% reliable, and these are rather different criteria

Actually, I have neither requirement.

You are correct that, in England, they focus on formal specifications. DoD/FAA
does not require flight control software in the U.S. to have a formal specification
(although DO-178 permits it.)

> >What I actually asked was, "Is there some way to modify the scope of the ACVC
> >process to improve compiler quality across all vendors? Or, is there something
> >outside the scope of the ACVC that could be done to improve compiler quality
> >across all vendors?"
> 
>   Not clear, the danger as I note above is that if you move in the wrong
>   direction here, you can easily damage compiler quality.

And, as far as I can tell, your opinion is that all directions are wrong. If
you care to refute this, you should in fairness indicate at least one _right_
direction.

Furthermore, you appear to find any attempt to make the issue less "Not clear"
to be anathema.

>   You missed the point of my :-) We expect GNAT to succeed, and to invest
>   substantial resources in improving its quality even if we don't get your
>   $25 million. There is no doubt that your $25 million check would have a
>   positive impact on GNAT, but we can manage without it :-)

> I notice you use the word "conformance" rather than "quality". Are these
> synonyms, to you? They aren't to me. I suspect they aren't to Mr. McCabe,
> or most other vendors.
> 
>   No of course they are not synonyms! That's the whole point.

Not _my_ whole point. I suspect Mr McCabe would disagree as well.
Let me know when you want to talk about quality.

> ACVC measures
>   conformance which is just one aspect of quality. What did I ever say that
>   made you think I think these are synonymous?

Specifically? Every time I ask about quality, you respond with arguments on
conformance. You say that conformance is an aspect of quality, yet here's
your list of where users have trouble:

     1. Safe compiler bugs (compiler bombs or gives an incorrect message)
     2. Unsafe compiler bugs (compiler generates wrong code)
     3. Performance is inadequate
     4. Optional features define     
     5. Features not defined in the RM are needed (bindings, libraries,
        preprocessors, other tool interfaces).
     6. Error messages are confusing
     7. Implementation dependent decisions are different from other
        compilers.
     8. Capacity limitations

Too bad there's no industry-wide efforts to tackle _these_....

>   The whole point of my
>   comments is that they are NOT synonymous.

Let me know if you ever decide to answer my questions.