V-22 Osprey and exception handling

V-22 Osprey and exception handling

[Ken Garlington]

I haven't seen the full JAGMAN report handed out at the Marine Corps
briefing on the latest V-22 Osprey crash (and if anyone has a link, please
let me know!). However, from what was said at the Marine Corps press
briefing a few days ago, it looks like we may have a no-kidding, post-test,
primary flight control software bug that led to a fatality. This breaks a
pretty impressive string of successes by the industry, IMO. I hope it's an
aberration, and not a foreshadowing of problems we're going to have in our
increasingly-complex implementations.

Speculating further, it looks like this may be another example of what
concerns me about the use of exception handling mechanisms in
safety-critical software: being able to accurately react to the failure
condition. Ariane 5 essentially shut down processing (bad idea, as it turned
out). V-22 did a system reset (pilot commanded, but it could have just as
easily been in response to an exception) and apparently introduced an
unexpected pitch transient. Such transients are always a danger when a
feedback system "starts over," and I know other fly-by-wire aircraft that
can give you a little "bump" in pitch after pushing the FLCS reset under
certain conditions.

If you're interested, a transcript of the press briefing is at

http://www.defenselink.mil/news/Apr2001/t04052001_t405mv22.html

Re: V-22 Osprey and exception handling

[pab49162]

After reviewing the briefing side on the primary flight control system (FCS)
software anomaly
(http://www.defenselink.mil/news/Apr2001/010405-D-6570C-003.jpg), I have a
couple comments I would like to share on this subject:

1.  In reviewing the information on the slide, I personally would disagree
that this was a "software" anomaly.  I strongly suspect that the software
operated exactly as the system requirement specified.  In other words, the
FCS software implemented the specified system-level design correctly.  The
root cause was most likely that the FCS requirements were incorrect and did
not correctly accommodate the hydraulics system first failure.

2. The use of  the term "reset" may be a little misleading to some people.
In most fly-by-wire flight control systems, the reset button (if it even
exists) does not do a "hard reset" such as the CTRL-ALT-DEL sequence on a
PC.  Rather, the reset signal is an input fed into various part of the
software to reset latched failures in the signal management voters/monitors
and to reset the control laws to some predetermined condition.

3. In most systems, the occurrence of such failures is accommodated by the
FCS design and does not constitute what I would consider to the appropriate
use of Ada's exception handling mechanism.  (In the design of a typical FCS
system, most of the effort is spent addressing the common "bad day"
scenarios in which there are one or more failures in the system.)  Normally,
I use exception handling for conditions that should "never" happen like
maybe a floating point overflow.  In an FCS, the accommodation of a
hydraulic failure does not fall into that category.

4. I would probably disagree with your comment "this breaks a pretty
impressive string of successes by the industry".  Over the last ten years or
so, there have been a fair number of incidents which were the direct result
of the errors in the system-level design of an air vehicle's flight control
system.  The ones that come immediately to mind include the crash of the
YF-22 prototype in 1992, the crash of the JAS-39 Gripen on flight 6 in 1989,
the crash of a second JAS-39 at an air show in 1993 and the crash of the
DarkStar unmanned aerial vehicle (UAV) on flight 2 in 1996.

As a final note, I should probably state that while I have 15+ years in
software development of fly-by-wire flight control systems for military
aircraft, I am not directly involved in the V-22 program.

Re: V-22 Osprey and exception handling

[Ken Garlington]

<pab49162@hotmail.com> wrote in message
news:djPz6.9$kb.47845@news1.i1.net...
: After reviewing the briefing side on the primary flight control system
(FCS)
: software anomaly
: (http://www.defenselink.mil/news/Apr2001/010405-D-6570C-003.jpg), I have a
: couple comments I would like to share on this subject:
:
: 1.  In reviewing the information on the slide, I personally would disagree
: that this was a "software" anomaly.  I strongly suspect that the software
: operated exactly as the system requirement specified.  In other words, the
: FCS software implemented the specified system-level design correctly.  The
: root cause was most likely that the FCS requirements were incorrect and
did
: not correctly accommodate the hydraulics system first failure.

Well, it's an interesting argument as to whether this is a "requirement" or
"design" failure; it depends upon how the requirements are specified. Since
this is probably not a function described in the control laws, I suspect the
requirements description was fairly general. Speculating further, I would
guess that when the reset button was pressed, some failed value was fed back
into the system

: 2. The use of  the term "reset" may be a little misleading to some people.
: In most fly-by-wire flight control systems, the reset button (if it even
: exists) does not do a "hard reset" such as the CTRL-ALT-DEL sequence on a
: PC.  Rather, the reset signal is an input fed into various part of the
: software to reset latched failures in the signal management
voters/monitors
: and to reset the control laws to some predetermined condition.

Actually, there can be levels of resets. For example, on the F-16, there is
a pilot reset which works somewhat as you describe (it removes one level of
latched failures, but does not send the control laws into fixed gains or
anything like that). There is also a "global reset" (not recommeded for
in-air use) that clears *all* latched failures. Finally, there is a DBU
switch that can be used similarly to a CTRL-ALT-DEL. It's not clear to me
which reset was involved here.

: 3. In most systems, the occurrence of such failures is accommodated by the
: FCS design and does not constitute what I would consider to the
appropriate
: use of Ada's exception handling mechanism.  (In the design of a typical
FCS
: system, most of the effort is spent addressing the common "bad day"
: scenarios in which there are one or more failures in the system.)
Normally,
: I use exception handling for conditions that should "never" happen like
: maybe a floating point overflow.  In an FCS, the accommodation of a
: hydraulic failure does not fall into that category.

I think my point was more along the lines that this is another case where a
function that was part of the failure handling requirements, actually
contributed to an accident instead.

So, what *do* you use in your Ada exception handlers, that adds more benefit
than risk?

: 4. I would probably disagree with your comment "this breaks a pretty
: impressive string of successes by the industry".  Over the last ten years
or
: so, there have been a fair number of incidents which were the direct
result
: of the errors in the system-level design of an air vehicle's flight
control
: system.
: The ones that come immediately to mind include the crash of the
: YF-22 prototype in 1992, the crash of the JAS-39 Gripen on flight 6 in
1989,
: the crash of a second JAS-39 at an air show in 1993 and the crash of the
: DarkStar unmanned aerial vehicle (UAV) on flight 2 in 1996.

All of these (except perhaps for the second JAS-39 crash, and I was thinking
that that one also fell into this category) happened while the aircraft was
still in a flight test program. I specifically excluded those from my answer
("post-test" being my exact words). Furthermore, IIRC, none of these "led to
a fatality", as I also noted. For that matter, I wouldn't even count the
YF-22 prototype as a software fault, since it was operated outside its
defined envelope.

The V-22, however, was a low-rate initial production (LRIP) configuration
accepted for use in August 2000, according to the briefing. It was presumed
to be a fully-tested article.

: As a final note, I should probably state that while I have 15+ years in
: software development of fly-by-wire flight control systems for military
: aircraft, I am not directly involved in the V-22 program.

16+ and me neither.

Re: V-22 Osprey and exception handling

[singlespeeder]

<pab49162@hotmail.com> wrote in message
news:djPz6.9$kb.47845@news1.i1.net...

> 4. I would probably disagree with your comment "this breaks a pretty
> impressive string of successes by the industry".  Over the last ten years
or
> so, there have been a fair number of incidents which were the direct
result
> of the errors in the system-level design of an air vehicle's flight
control
> system.  The ones that come immediately to mind include the crash of the
> YF-22 prototype in 1992, the crash of the JAS-39 Gripen on flight 6 in
1989,
> the crash of a second JAS-39 at an air show in 1993 and the crash of the
> DarkStar unmanned aerial vehicle (UAV) on flight 2 in 1996.
>

or the FADEC failure that allegedly brought down the Chinook on the Mull of
Kintyre in 1994.
it's not just FCS software that can bring an air vehicle down...

singlespeeder

Re: V-22 Osprey and exception handling

[Ken Garlington]

"singlespeeder" <singlespeeder@32sixteen.com> wrote in message
news:9aq517$79g$1@neptunium.btinternet.com...
:
: <pab49162@hotmail.com> wrote in message
: news:djPz6.9$kb.47845@news1.i1.net...
:
: > 4. I would probably disagree with your comment "this breaks a pretty
: > impressive string of successes by the industry".  Over the last ten
years
: or
: > so, there have been a fair number of incidents which were the direct
: result
: > of the errors in the system-level design of an air vehicle's flight
: control
: > system.  The ones that come immediately to mind include the crash of the
: > YF-22 prototype in 1992, the crash of the JAS-39 Gripen on flight 6 in
: 1989,
: > the crash of a second JAS-39 at an air show in 1993 and the crash of the
: > DarkStar unmanned aerial vehicle (UAV) on flight 2 in 1996.
: >
:
: or the FADEC failure that allegedly brought down the Chinook on the Mull
of
: Kintyre in 1994.

Of course, it has not been definitively established that there _was_ a FADEC
failure related to this accident, only speculation as to the possibility
AFAIK. See http://www.cw360.com for more information (search for "Chinook").
Therefore, it's premature to include this as an example. (I exclude this by
using the term "no-kidding" in my full statement, which was:

"...it looks like we may have a no-kidding, post-test, primary flight
control software bug that led to a fatality. This breaks a pretty impressive
string of successes by the industry, IMO."

Re: V-22 Osprey and exception handling

[Ted Dennison]

In article <1Q_z6.474$VI4.153258056@newssvr10-int.news.prodigy.com>, Ken
Garlington says...
>
><pab49162@hotmail.com> wrote in message
>news:djPz6.9$kb.47845@news1.i1.net...
>: 3. In most systems, the occurrence of such failures is accommodated by the
>: FCS design and does not constitute what I would consider to the
>appropriate
>: use of Ada's exception handling mechanism.  (In the design of a typical
..
>: I use exception handling for conditions that should "never" happen like
>: maybe a floating point overflow.  In an FCS, the accommodation of a
>: hydraulic failure does not fall into that category.
>
>I think my point was more along the lines that this is another case where a
>function that was part of the failure handling requirements, actually
>contributed to an accident instead.

There really isn't enough information in the report to even go that far. For all
we know, it could have been caused by code that was *not* handling possible
failures when it should have been. Until the other report that is to go into the
aircraft itself comes out, its just too premature to say (for all we know, that
report might go completely against this one, and blame "pilot error"). It
certainly *way* premature to be talking about specific language features.

---
T.E.D.    homepage   - http://www.telepath.com/dennison/Ted/TED.html
          home email - mailto:dennison@telepath.com

Re: V-22 Osprey and exception handling

[Ken Garlington]

Ted Dennison" <dennison@telepath.com> wrote in message
news:knjA6.787$FY5.40878@www.newsranger.com...
: In article <1Q_z6.474$VI4.153258056@newssvr10-int.news.prodigy.com>, Ken
: Garlington says...
: >
: ><pab49162@hotmail.com> wrote in message
: >news:djPz6.9$kb.47845@news1.i1.net...
: >: 3. In most systems, the occurrence of such failures is accommodated by
the
: >: FCS design and does not constitute what I would consider to the
: >appropriate
: >: use of Ada's exception handling mechanism.  (In the design of a typical
: ..
: >: I use exception handling for conditions that should "never" happen like
: >: maybe a floating point overflow.  In an FCS, the accommodation of a
: >: hydraulic failure does not fall into that category.
: >
: >I think my point was more along the lines that this is another case where
a
: >function that was part of the failure handling requirements, actually
: >contributed to an accident instead.
:
: There really isn't enough information in the report to even go that far.
For all
: we know, it could have been caused by code that was *not* handling
possible
: failures when it should have been. Until the other report that is to go
into the
: aircraft itself comes out...

Which report is this?

: its just too premature to say (for all we know, that
: report might go completely against this one, and blame "pilot error"). It
: certainly *way* premature to be talking about specific language features.

I think you've missed the point. I'm not suggesting that a particular
language feature caused the V-22 crash. Please re-read the original post.

: ---
: T.E.D.    homepage   - http://www.telepath.com/dennison/Ted/TED.html
:           home email - mailto:dennison@telepath.com

Re: V-22 Osprey and exception handling

[Ted Dennison]

In article <3htA6.809$j53.240510015@newssvr10-int.news.prodigy.com>, Ken
Garlington says...
>
>
>Ted Dennison" <dennison@telepath.com> wrote in message
>news:knjA6.787$FY5.40878@www.newsranger.com...
>: In article <1Q_z6.474$VI4.153258056@newssvr10-int.news.prodigy.com>, Ken
>: Garlington says...
>: failures when it should have been. Until the other report that is to go
>: >into the aircraft itself comes out...
>
>Which report is this?

From the press conference that you posted:
----
Second, I would like to point out that the purpose of this brief is limited to
providing you information about the JAG Manual investigation.
..
Now, a JAG investigation is conducted to determine the cause of a mishap. We
also use it to identify lapses or shortcomings in processes and procedures; to
direct corrective actions; and to adjudicate claims against the government.
Finally, it can serve as the basis of administrative or legal actions. 
..
The JAG investigation should not be confused with the Aircraft Mishap Board
investigation that was conducted on the same accident. The Aircraft Mishap Board
investigation, which focuses specifically on aviation safety and accident
prevention, is in the review process. 

----

There's also:

----
As I mentioned earlier, the investigation revealed an anomaly in the computer
software that governs control of the prop rotors. This anomaly rendered the
emergency procedures outlined in the Naval Air Training and Operations
Procedures Standardization, or NATOPS, flight manual ineffective. As a result,
this investigation recommends, and Headquarters United States Marine Corps
concurs, that Naval Air Systems Command, NAVAIR, conduct a complete and
comprehensive review of the entire vehicle management system, the computer
software, to identify design deficiencies that may exist in any logic path that
could adversely affect aircraft controlability. 
----

In other words, there was clearly (to their minds) some kind of problem when
that acknowledge button is pressed (as per procedure) during this one particular
circumstance. They did not see it as their job to track this problem down any
further (JAG is just for seeing if their procedures are bad, or their personnel
were negligent in some way), so they are asking for a separate review to get to
the bottom of it.

At least, that's the way I read it.

---
T.E.D.    homepage   - http://www.telepath.com/dennison/Ted/TED.html
          home email - mailto:dennison@telepath.com

Re: V-22 Osprey and exception handling - warning very little ada content

[singlespeeder]

"Ken Garlington" <Ken.Garlington@computer.org> wrote in message
news:Ot8A6.553$KE3.180353065@newssvr10-> : or the FADEC failure that
allegedly brought down the Chinook on the Mull
> of
> : Kintyre in 1994.
>
> Of course, it has not been definitively established that there _was_ a
FADEC
> failure related to this accident, only speculation as to the possibility
> AFAIK

That's why I said "allegedly".

> See http://www.cw360.com for more information (search for "Chinook")

For a more intriguing and paranoid explanation try searching for "Chinook "
and "Aurora".

On the topic of black projects I note that one of the reasons given for the
short development timescales of the F-117 was the lack of external auditing.
Perhaps the influence of Ada on developing robust software was underplayed,
as apparently the F-117 has the aerodynamics of a fridge-freezer which must
make for an "interesting" FCS.

Nick

Nick

Re: V-22 Osprey and exception handling - warning very little ada content

[Ken Garlington]

"singlespeeder" <singlespeeder@32sixteen.com> wrote in message
news:9avon9$874$1@plutonium.btinternet.com...
:
: On the topic of black projects I note that one of the reasons given for
the
: short development timescales of the F-117 was the lack of external
auditing.
: Perhaps the influence of Ada on developing robust software was
underplayed,
: as apparently the F-117 has the aerodynamics of a fridge-freezer which
must
: make for an "interesting" FCS.

The aerodynamics of the F-117 are wholly unrelated to the robustness of the
FCS software. The F-117 is a ground attack aircraft; it doesn't need the
manueverability of a fighter.

Re: V-22 Osprey and exception handling

[Ken Garlington]

"Ted Dennison" <dennison@telepath.com> wrote in message
news:L7GA6.2062$FY5.147618@www.newsranger.com...
: In article <3htA6.809$j53.240510015@newssvr10-int.news.prodigy.com>, Ken
: Garlington says...
: >
: >
: >Ted Dennison" <dennison@telepath.com> wrote in message
: >news:knjA6.787$FY5.40878@www.newsranger.com...
: >: In article <1Q_z6.474$VI4.153258056@newssvr10-int.news.prodigy.com>,
Ken
: >: Garlington says...
: >: failures when it should have been. Until the other report that is to go
: >: >into the aircraft itself comes out...
: >
: >Which report is this?
:
: From the press conference that you posted: [Aircraft Mishap Board]

AFAIK, military mishap board results are not released to the public.
Furthermore, as the General noted, the mishap board's primary focus is on
developing recommendations for avoiding future mishaps. The JAGMAN report is
intended to focus on the cause of a mishap. I would not expect the mishap
board to generate much more information with respect to the cause of the
accident.

With respect to the NAVAIR review, this will also be focused on making sure
there are no similar failure paths in the software, not further identifying
the root cause of the accident.

I recently had a chance to get more information from the company that
developed the software, and the root cause of this particular incident
apparently is well understood at this point.