Re: Trust but verify (was Re: Papers on the Ariane-5 crash and Design by Contract

[eachus@spectre.mitre.org (Robert I. Eachus)]

In article <3338AA0A.7A79CB24@eiffel.com> Bertrand Meyer <bertrand@eiffel.com> writes:

  > But of course this is not a reason to say there should not be
  > simulation, testing, inspections, independent reviews etc. etc.
  > I don't recall anyone even coming close to suggesting this in the
  > present thread. And our paper certainly does not say anything of
  > the sort.

  But the Final Report on the Ariane 5 failure does say that there
were NO simulation, testing, inspections, independent reviews
etc. etc.  for the Ariane 5.  Not minimal, not inadequate, none.  And
this is where your paper misleads.  Any "reasonable" methods would be
better than no testing, no inspections, and no reviews.

  > It would be as absurd to use Design by Contract or other a priori
  > relability techniques as an excuse not to do a posteriori V & V
  > (testing, simulation etc.), as it would be to develop in a sloppy
  > way with the expectation that testing will uncover any problems.

  But in this case, effectively both excuses were used.  No
requirements review was done because there would be full up testing,
then the full-up test was cancelled because the software had not
changed!

  > But the need for a posteriori verification is too often an excuse
  > for not applying the proven a priori reliability techniques, such
  > as Design by Contract. This is not only silly but a recipe for
  > disaster of Ariane-5 proportions.

    Yep, and doing neither is not a recipe, it is a guarantee.

  > I think that its arguments and conclusion are perfectly in line with
  > many useful comments made by people who thought they disagreed with it.
  > Please check what it really has to say.

    What Bertrand Meyer continues to miss, probably because it is so
difficult to believe, is that the software (and hardware) were reused
without any requirements review or testing.  None.  Not inadequate
review or incomplete testing, none.  His paper assumes that review
and testing occured but missed a subtle point.  There were several
glaringly obvious problems that would not have been missed by any
competent engineer--they probably wouldn't have been missed by a
teenager whose only experience was in model rocketry:

    1) The alignment software continued running for 40 seconds after
launch, for reasons documented in the code but totally irrelevant to
Ariane 5.

    2) The permissible engine deflection angles were based on Ariane 4
requirements, and could (and did) result in destruction of an Ariane
5.  (Would lesser max deflection values have met Ariane 5 requirements
for crosswind tolerance and wind shear?  No clue, the analysis wasn't
done.)

    3) There were several other analyses that depended on knowledge of
Ariane 4 flight paths and physical limits.  Again these were very
invalid but not redone.  (This includes the failure that triggered the
entire chain of events.)

    4) The assumption--well documented--in the code that any Ada
exception not locally handled was due to hardware not software
failure.  In Ariane 4, this assumption may be correct.  But in Ariane
5, the decision should have been reconsidered, at least for the first
test flight.

    I was determined that I had made my last post on this subject, but
I got sucked in again.  I want to make sure that no one misunderstands
the nature of this failure--it was a failure of management, not of
software engineering.  And it wasn't even software management that
failed, the politics that caused the failure occured at a much higher
management level.

   In this it was very similar to the Apollo fire and the Challenger
disaster.  In the Apollo fire, two tests with very different
requirements were merged to "speed things up."  Three psi oxygen and
three psi above ambient atmosphere are two very different things.  But
the tests were merged after the safety engineers had signed off on
them separately.

   George Low was brought in to take over the Apollo program after the
fire, and he instituted procedures to prevent anything similar from
ever happening again on a manned test or manned flight.  These
procedures were followed until STS 25.  (Although there were a number
of leaks of "pressure to maintain the launch schedule" heard before
then, the final flight worthiness decision for the shuttle and all
subsystems was always made by an engineer.)

--

					Robert I. Eachus

with Standard_Disclaimer;
use  Standard_Disclaimer;
function Message (Text: in Clever_Ideas) return Better_Ideas is...