Re: Critique of Ariane 5 paper (finally!)

[Nick Leaton <nickle@calfp.co.uk>]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 8512 bytes --]


Ken Garlington wrote:
> 
> Nick Leaton wrote:
> >
> > Let us say for the moment that in some circumstances DBC helps.
> > For those that have been critising DBC, since DBC is optional, and is an
> > addition to the current methodology I think the only valid criticism one
> > make, is if you can find a situation where DBC causes a problem.
> >
> > That is, where does DBC screw up, and the current methodologies do not.
> 
> See my Ariane paper for some examples of such information:
> 
>   http://www.flash.net/~kennieg/ariane.html#s3.1.6

QUOTE -----------------------------------------------------------
3.1.6 Adverse effects of documenting assertions

     There is a line of reasoning that says, "Even if DBC/assertions
would have been of minimal use, why not include them anyway just in case
they do
     some good?" Such a statement assumes there are no adverse impacts
to including assertions in code for documentation purposes. However,
there
     are always adverse effects to including additional code: 

         As with any other project, there are management pressures to
meet cost and schedule drivers. Additional effort, therefore, is
discouraged
         unless justified.

         More importantly, all projects have finite time and other
resources available. Time spent on writing and analyzing assertions is
time not
         spent elsewhere. If the work that is not performed as a result
would have been more valuable (in terms of its effect on the safety and
         integrity of the system) than the time spent with assertions,
then the resulting system may be less safe. 

     There is a growing consensus in the safety-critical software field
that simpler software tends to be safer software [21]. With this
philosophy,
     additional code such as assertions should only be added where there
is a clear benefit to the overall safety of the particular system being
     developed. 
END QUOTE --------------------------------------------------------

1) Additional effort.
There is plenty of evidence that the cost of finding and fixing a bug
early in the development process is much cheaper that fixing it later.
Now *IF* DBC as a methodology produces earlier detection of faults, then
this 'additional effort' is justified, as it is effort early in the
development process. I believe this to be the case from practical
experience. Developing code last week I detected errors in my code very
early, even though the fault would have only shown itself in an
exception. I had an invariant that made sure I had an error handler in a
class. It broke as soon as I tried to construct the class, it didn't
wait until I tried to generate an error, all because I hadn't set the
handler up. That combined with permantant testing of assertions has the
effect of producing less effort.

2) Time spent else where.
Is this the case? Some of it may be, but I believe if you cannot be
rigourous about what your software is trying to do, by writing
assertions, then you are unlikely to produce a quality system. The
effect of writing assertions overlaps with the design process. It is not
wasted time, it just comes under a different heading. If your design
process listed the assertions in the specification, would implementing
them be a waste of effort?

3) Simple software. 
You bet. The simpler the better. Occams Razor rules. Now here there is a
split between DBC a la Eiffel and DBC, say in C++. In Eiffel it is
simple. In C++ it is hard, particularly with inheritance of assertions.
One common theme from Eiffel practitioners is their support for DBC.
Why? They are simple to write. Prior to using Eiffel I had read about
the language. I was extremely sceptical about assertions because in my
experience with C++ and C++ programmers, no one writes them, mainly
because it is hassle. Take away the hassle and people will write them
because of the benefits.



>   http://www.flash.net/~kennieg/ariane.html#s3.2.2

QUOTE --------------------------------------------------------
3.2.2 Adverse effects of testing with assertions

     Assume for a moment that the proper testing environment and data
had been available. Putting aside for the moment the question as to
whether
     assertions would have been necessary to detect the fault (see
section 4.2), are there any disadvantages to using assertions during
testing, then
     disabling them for the operational system? In the author's
experience, there are some concerns about using this approach for
safety-critical
     systems: 

         The addition of code at the object level obviously affects the
time it takes for an object to complete execution. Particularly for
real-time
         systems (such as the Ariane IRS), differences in timing between
the system being tested and the operational system may cause timing
         faults, such as race conditions or deadlock, to go undetected.
Such timing faults are serious failures in real-time systems, and a test
which
         is hindered from detected them loses some of its effectiveness.

         In addition, the differences in object code between the tested
and operational systems raise the issue of errors in the object code for
the
         operational system. Such errors are most likely to occur due to
an error in the compilation environment, although it is possible that
other
         factors, such as human error (e.g. specifying the wrong version
of a file when the code is recompiled) can be involved. For example, the
         author has documented cases where Ada compilers generate the
correct code when exceptions are not suppressed, but generate
         incorrect code (beyond the language's definition of
"erroneous") when they are suppressed. This is not entirely unexpected;
given the
         number of user-selectable options present with most compilation
environments, it is difficult if not impossible to perform adequate
toolset
         testing over all combinations of options. Nothing in the Eiffel
paper indicates that Eiffel compilers are any better (or worse) than
other
         compilers in this area. Although this is a fault of the
implementation, not the language or methodology, it is nonetheless a
practical limitation
         for safety-critical systems, where one object code error can
have devastating results. 

     One possible approach to resolving this issue is to completely test
the system twice; once with assertions on and another time with
assertions
     suppressed. However, the adverse factors described in section 3.1.6
then come into play: By adding to the test time in this manner, other
useful
     test techniques (which might have greater value) are not performed.
Generally, it is difficult to completely test such systems once, never
mind
     twice! This effect is worse for safety-critical systems that
perform object-code branch coverage testing, since this testing is
completely
     invalidated when the object code changes [25]. 

     Overall, there are significant limitations to using this technique
for safety-critical systems, and in particular real-time systems such as
the Ariane
     5 IRS. 

END QUOTE --------------------------------------------------------

Assertions affect timing in safety critical systems.

Firstly it depends on the implementation. It is easy to envisage a
system where the assertions are redundantly executed. But you would only
want to do that if you were running with faulty software ?!*^�%

I also find it worrying that systems are being used for safety critical
apps where there is the possibility of a race or deadlock to occur. 

Compilation problems.
These can occur in any system as you are aware. From discussions with
some of the people involved in writting Eiffel compilers, the enabling,
disabling of assertions has a very trivial implementation, which is very
unlikely to go wrong. It has also be extensively tested in the field by
end users.
Do you trust your compiler? If not, you shouldn't be writing safety
critical software with it. Period.

Next I find some of the logic of your arguments here very weak.
Paraphrasing. We have trouble testing safety critical systems, but we
will use them anyway. Hmmm.


>   http://www.flash.net/~kennieg/ariane.html#s3.3
> 
> There are approaches that can avoid such costs, particularly those of
> 3.2.2 and 3.3 (by not requiring object code modification). 3.1.6 can
> be mitigated through the use of techniques that minimize cost (e.g.
> automated structural testing analysis).
> 
> >
> > --
> >
> > Nick

-- 

Nick