Re: Critique of Ariane 5 paper (finally!)

[Samuel Mize <smize@link.com>]

Robert S. White wrote:
>   The other issue is that in the Ariane 5 case, the
> methodology that was in place (system requirements review and
> software requirements specification), was not followed
> adequately.  To quote the inquiry report once more:
...
>   My complaint is against the claim in the Eiffel paper:
> 
>   "Does this mean that the [Ariane 5] crash would
>   automatically have been avoided had the mission used
>   a language and method supporting built-in assertions
>   and Design by Contract? Although it is always risky
>   to draw such after-the-fact conclusions, the answer
>   is probably yes..."
>      ^^^^^^^^^^^^
> 
>     I say, IMO, probably no for the Ariane 5 case.

I'd even tolerate the "probably yes," if it weren't
explicitly stated that DBC is the ONLY method that would
probably have avoided the crash.

No discussion about whether DBC would have helped is relevant
to my point.  I concede that DBC is one method that would
have helped.

I'll say that again:

    I CONCEDE THAT DBC WOULD PROBABLY HAVE HELPED, IF ONLY
    BECAUSE IT ISN'T DBC WITHOUT REVIEWS AND TEST.

But the paper says that ONLY DBC would have helped, as I'll
show below.  If the authors had said "Yes, the claim was too
strong, sorry," a lot of us would have shut up and gone away.

Co-author Jean-Marc Jezequel has said that this does not
characterize what the paper was MEANT to say.  I have
little dispute with what he says they MEANT to say[1]:

  Let's finally sum up what I perceive as the most important
  claims in this paper:
  - reusing a component without checking its full specification
    is dangerous, which means that simple minded CORBA-like
    approaches at building components for mission-critical
    software are doomed.
  - using design by contract is an interesting way to specify the
    behavior of a component
  - at least in the case of Ariane 501, simple assertions (a la
    Eiffel and other languages) would have been expressive enough
    to specify the fatal hidden assumption.

But the paper DOES state explicitly that DBC is the ONLY method
that would have avoided the crash, and that existing methods were
applied but did not succeed.

Following are my reasons for stating that the paper says this.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
THE PAPER CLAIMS THAT ARIANE 5 APPLIED ALL RELEVANT EXISTING METHODS

At the very least it strongly implies this.

From the paper:
    The ESA's software people knew what they were doing and applied
    widely accepted industry practices.

No, the project made an explicit decision to NOT apply widely
accepted industry practices.

I have yet to see any support for this assertion: not in the Eiffel
paper, not in the ESA report, not in the net traffic.


From the paper:
    Is it a testing error?  Not really.  ...  But if one can test more
    one cannot test all.  Testing, we all know, can show the presence
    of errors, not their absence.

This implies that the error in question would not likely have been
found through normal testing.  Yet it is, in fact, one that would
have blown out a normal test scenario the first time they tried it.
The addition of DBC would have made no more difference than would
the addition of paper hats.  There may indeed be errors that cannot
be found through testing, and that DBC would find, but this is NOT
demonstrated by the Ariane 5 case.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
THE PAPER CLAIMS THAT ONLY DBC WOULD HAVE PREVENTED THIS ERROR

The concluding section, labelled "The lesson for every software
developer," is clearly meant to state what the Ariane 5 crash shows.
It states:

    To attempt to reuse software without Eiffel-like assertions is
    to invite failures of potentially disastrous consequences.
    ...
    For reuse to be effective, Design by Contract is a requirement.
    Without a precise specification attached to each reusable
    component -- precondition, postcondition, invariant -- no one
    can trust a supposedly reusable component.

This says that, if a reused component has not been analyzed with
DBC, you cannot trust it, and you are inviting failure.  No other
method is sufficient, by this statement.


From the paper:
    Reuse without a contract is sheer folly.

Does this state that Ariane 5 would have crashed, no matter what
other methods were used, unless DBC were also used?  In context,
yes.  It's from the conclusions section, "The lesson for every
software developer."  Surely this is meant to be the lesson of the
Ariane 5 crash.  Surely "contract" in this context is intended to
refer specifically to a Design By Contract artifact.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[1] I'd dispute his first point if "full specification" is limited
    to pre/post conditions and invariants.  In some cases these are
    too little, in others they may not be needed.  However, it's
    certainly failure-prone to reuse components without reviewing
    their specs and designs.  I know too little about CORBA to have
    an opinion on its (in)sufficiency.

Sam Mize