From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=0.5 required=5.0 tests=BAYES_00, PP_MIME_FAKE_ASCII_TEXT,REPLYTO_WITHOUT_TO_CC autolearn=no autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII X-Google-Thread: fac41,f66d11aeda114c52 X-Google-Attributes: gidfac41,public X-Google-Thread: 103376,f66d11aeda114c52 X-Google-Attributes: gid103376,public From: Nick Leaton Subject: Re: Critique of Ariane 5 paper (finally!) Date: 1997/08/26 Message-ID: <3402A529.CCFDAC8C@calfp.co.uk> X-Deja-AN: 268581339 X-NNTP-Posting-Host: calfp.demon.co.uk [158.152.70.168] References: <872172435.980@dejanews.com> <33FC66AD.9A0799D4@calfp.co.uk> <33FFA4B1.3543@flash.net> Reply-To: nickle@pauillac Newsgroups: comp.lang.ada,comp.lang.eiffel Date: 1997-08-26T00:00:00+00:00 List-Id: Ken Garlington wrote: > > Nick Leaton wrote: > > > > Let us say for the moment that in some circumstances DBC helps. > > For those that have been critising DBC, since DBC is optional, and is an > > addition to the current methodology I think the only valid criticism one > > make, is if you can find a situation where DBC causes a problem. > > > > That is, where does DBC screw up, and the current methodologies do not. > > See my Ariane paper for some examples of such information: > > http://www.flash.net/~kennieg/ariane.html#s3.1.6 QUOTE ----------------------------------------------------------- 3.1.6 Adverse effects of documenting assertions There is a line of reasoning that says, "Even if DBC/assertions would have been of minimal use, why not include them anyway just in case they do some good?" Such a statement assumes there are no adverse impacts to including assertions in code for documentation purposes. However, there are always adverse effects to including additional code: As with any other project, there are management pressures to meet cost and schedule drivers. Additional effort, therefore, is discouraged unless justified. More importantly, all projects have finite time and other resources available. Time spent on writing and analyzing assertions is time not spent elsewhere. If the work that is not performed as a result would have been more valuable (in terms of its effect on the safety and integrity of the system) than the time spent with assertions, then the resulting system may be less safe. There is a growing consensus in the safety-critical software field that simpler software tends to be safer software [21]. With this philosophy, additional code such as assertions should only be added where there is a clear benefit to the overall safety of the particular system being developed. END QUOTE -------------------------------------------------------- 1) Additional effort. There is plenty of evidence that the cost of finding and fixing a bug early in the development process is much cheaper that fixing it later. Now *IF* DBC as a methodology produces earlier detection of faults, then this 'additional effort' is justified, as it is effort early in the development process. I believe this to be the case from practical experience. Developing code last week I detected errors in my code very early, even though the fault would have only shown itself in an exception. I had an invariant that made sure I had an error handler in a class. It broke as soon as I tried to construct the class, it didn't wait until I tried to generate an error, all because I hadn't set the handler up. That combined with permantant testing of assertions has the effect of producing less effort. 2) Time spent else where. Is this the case? Some of it may be, but I believe if you cannot be rigourous about what your software is trying to do, by writing assertions, then you are unlikely to produce a quality system. The effect of writing assertions overlaps with the design process. It is not wasted time, it just comes under a different heading. If your design process listed the assertions in the specification, would implementing them be a waste of effort? 3) Simple software. You bet. The simpler the better. Occams Razor rules. Now here there is a split between DBC a la Eiffel and DBC, say in C++. In Eiffel it is simple. In C++ it is hard, particularly with inheritance of assertions. One common theme from Eiffel practitioners is their support for DBC. Why? They are simple to write. Prior to using Eiffel I had read about the language. I was extremely sceptical about assertions because in my experience with C++ and C++ programmers, no one writes them, mainly because it is hassle. Take away the hassle and people will write them because of the benefits. > http://www.flash.net/~kennieg/ariane.html#s3.2.2 QUOTE -------------------------------------------------------- 3.2.2 Adverse effects of testing with assertions Assume for a moment that the proper testing environment and data had been available. Putting aside for the moment the question as to whether assertions would have been necessary to detect the fault (see section 4.2), are there any disadvantages to using assertions during testing, then disabling them for the operational system? In the author's experience, there are some concerns about using this approach for safety-critical systems: The addition of code at the object level obviously affects the time it takes for an object to complete execution. Particularly for real-time systems (such as the Ariane IRS), differences in timing between the system being tested and the operational system may cause timing faults, such as race conditions or deadlock, to go undetected. Such timing faults are serious failures in real-time systems, and a test which is hindered from detected them loses some of its effectiveness. In addition, the differences in object code between the tested and operational systems raise the issue of errors in the object code for the operational system. Such errors are most likely to occur due to an error in the compilation environment, although it is possible that other factors, such as human error (e.g. specifying the wrong version of a file when the code is recompiled) can be involved. For example, the author has documented cases where Ada compilers generate the correct code when exceptions are not suppressed, but generate incorrect code (beyond the language's definition of "erroneous") when they are suppressed. This is not entirely unexpected; given the number of user-selectable options present with most compilation environments, it is difficult if not impossible to perform adequate toolset testing over all combinations of options. Nothing in the Eiffel paper indicates that Eiffel compilers are any better (or worse) than other compilers in this area. Although this is a fault of the implementation, not the language or methodology, it is nonetheless a practical limitation for safety-critical systems, where one object code error can have devastating results. One possible approach to resolving this issue is to completely test the system twice; once with assertions on and another time with assertions suppressed. However, the adverse factors described in section 3.1.6 then come into play: By adding to the test time in this manner, other useful test techniques (which might have greater value) are not performed. Generally, it is difficult to completely test such systems once, never mind twice! This effect is worse for safety-critical systems that perform object-code branch coverage testing, since this testing is completely invalidated when the object code changes [25]. Overall, there are significant limitations to using this technique for safety-critical systems, and in particular real-time systems such as the Ariane 5 IRS. END QUOTE -------------------------------------------------------- Assertions affect timing in safety critical systems. Firstly it depends on the implementation. It is easy to envisage a system where the assertions are redundantly executed. But you would only want to do that if you were running with faulty software ?!*^�% I also find it worrying that systems are being used for safety critical apps where there is the possibility of a race or deadlock to occur. Compilation problems. These can occur in any system as you are aware. From discussions with some of the people involved in writting Eiffel compilers, the enabling, disabling of assertions has a very trivial implementation, which is very unlikely to go wrong. It has also be extensively tested in the field by end users. Do you trust your compiler? If not, you shouldn't be writing safety critical software with it. Period. Next I find some of the logic of your arguments here very weak. Paraphrasing. We have trouble testing safety critical systems, but we will use them anyway. Hmmm. > http://www.flash.net/~kennieg/ariane.html#s3.3 > > There are approaches that can avoid such costs, particularly those of > 3.2.2 and 3.3 (by not requiring object code modification). 3.1.6 can > be mitigated through the use of techniques that minimize cost (e.g. > automated structural testing analysis). > > > > > -- > > > > Nick -- Nick