Re: Martian Rover Spirit's mulitasking software reset.

[Paul Anderson <notme@nowhere.com>]

All:

>>>>The mars rover Spirit suffered a software reset as a result of a
>>>>multitasking problem.
>>>>"The apparent cause was that the rover's central processing unit was
>>>>overworked as several tasks were running in parallel at the time."
>>>> The rover uses VX works and I believe C. Testing obviously did not
>>>>catch this fault. Would the use of Ada  made any difference?
>>>>
>>>
>>>My understanding is that there was a "priority inverstion", so had
>>
>>That is also my understanding.

This isn't right.  The problem with Spirit was with the flash
memory.  It is hard to find a good description of exactly what
went wrong, but the reboots happened because there were too
many files open.  Whether Ada might have helped is really
impossible to say, given the paucity of information available.

The Mars Pathfinder mission from 1997 was the one that had the
priority inversion problem.  I doubt that they were aware of
the problem before launch, as it turned out to be so easy
to fix.  In VxWorks, when you create a semaphore you do
something like:

   sem_id = semMCreate(options);

where options is a bitmask that specifies the set of options
you want for the semaphore.  VxWorks implements the priority
inheritance protocol, so if you want to use it, you would do:

   sem_id = semMCreate(SEM_Q_PRIORITY | SEM_INVERSION_SAFE);

On the Pathfinder, they simply did not specify the
SEM_INVERSION_SAFE option.

Fortunately, the options were held in a global variable, and
they were able to upload a patch that updated that variable to
specify that option.

I don't believe that VxWorks had an implementation of the
priority ceiling protocol at that time.

For more information, see:
http://en.wikipedia.org/wiki/Priority_inversion
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html

There was a third Mars mission that had a software problem: the Mars
Polar Lander.  In this case, during entry to the atmosphere, a sensor
generated a spurious signal indicating that the craft had landed, and
the software set a bit accordingly.  This shut down the descent engines
prematurely.  The craft then plummetted to the surface and was
destroyed.  The software should have detected that the sensor input
was invalid at that time, and unset the problematic bit.  Apparently
the addition of a single line of code would have prevented the problem.

It is hard to see how Ada could have helped here either.

Paul