A new name for software failure : the glitch

A new name for software failure : the glitch

[francois_fabien]

In finance, software failure can be very expensive very fast.
http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/

Re: A new name for software failure : the glitch

[Maciej Sobczak]

W dniu sobota, 4 sierpnia 2012 20:18:21 UTC+2 użytkownik (Nieznane) napisał:

> In finance, software failure can be very expensive very fast.

Interestingly, this can also mean that there is something fundamentally broken with the way we do "finance" today. If you have a broken structure, then a genuine glitch in one place can turn the whole into dust. It's still a glitch and the actual problem is in the structure - the glitch can be merely a trigger for the collapse and the same glitch could be much less catastrophic if it happens in a structure that is in overall more resilient.

Note: I'm not trying to justify poor-quality software. I just think that this particular case shows that the problem of responsibility is more complex and cannot be reasoned about by just looking at a single piece of the puzzle - it is a *system* problem. The structure of "finance" today is just a shit waiting to hit the fan and the responsibility for this state is certainly not on the software side alone.

(Now ducking away to write my trading algos in Ada. ;-) )

-- 
Maciej Sobczak * http://www.msobczak.com * http://www.inspirel.com

Re: A new name for software failure : the glitch

[F. Fabien]

Le samedi 4 août 2012 21:58:01 UTC+2, Maciej Sobczak a écrit :
> W dniu sobota, 4 sierpnia 2012 20:18:21 UTC+2 użytkownik (Nieznane) napisał:

> Interestingly, this can also mean that there is something fundamentally broken with the way we do "finance" today. 

Finance can and has gone mad many times in history. The first and most famous
example is Tulip mania http://en.wikipedia.org/wiki/Tulip_mania

> (Now ducking away to write my trading algos in Ada. ;-) )
> 
The point is to know whether the software had a bad decision logic (a.k.a algorithm) or a good decision logic with faulty implementation.
In the second case, there is a market for sound software engineering (Ada)
in the finance sector.
But since banking is much more secretive than engineering, I doubt informations
will be publicly available on this failure.

Re: A new name for software failure : the glitch

[Dmitry A. Kazakov]

On Sat, 4 Aug 2012 12:58:01 -0700 (PDT), Maciej Sobczak wrote:

> W dniu sobota, 4 sierpnia 2012 20:18:21 UTC+2 u�ytkownik (Nieznane) napisa�:
> 
>> In finance, software failure can be very expensive very fast.
> 
> Interestingly, this can also mean that there is something fundamentally
> broken with the way we do "finance" today. If you have a broken structure,
> then a genuine glitch in one place can turn the whole into dust. It's
> still a glitch and the actual problem is in the structure - the glitch can
> be merely a trigger for the collapse and the same glitch could be much
> less catastrophic if it happens in a structure that is in overall more
> resilient.

I am not an expert in software for trading stocks. But my guess is that
there is nothing wrong with it. It looks like an automated system. What you
described can be summarized in one word: instability. Yes, it is instable,
considering code variations as the input and functional behavior as the
output. All software systems are. Which is a problem far more important
than fabulous global warming etc.

> (Now ducking away to write my trading algos in Ada. ;-) )

Ada is not so important as the attitude to the language and software design
it promotes.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: A new name for software failure : the glitch

[Georg Bauhaus]

On 05.08.12 09:41, Dmitry A. Kazakov wrote:
>> (Now ducking away to write my trading algos in Ada.;-)  )
> Ada is not so important as the attitude to the language and software design
> it promotes.

Regarding this case, I have heard that NY Stock Exchange
had introduced a new way of supplying liquidity to which
Knight Capital's system had somehow reacted in the wrong
direction.

A possible speculation about the cause of this failure,
then, involves that a piece in the system had been replaced
without performing a comprehensive check of the contracts.

That is, one piece had either continued trading with outdated
assumptions driving it, or had started trading with up to date,
but wrong assumptions about the new contracts that would now
apply.

Re: A new name for software failure : the glitch

[Britt]

Another "glitch": Southwest Airlines has a 50% off airfare sale but then charges credit cards 6 to 36 times over:

http://news.blogs.cnn.com/2012/08/04/southwest-airlines-promo-glitch-charges-customers-for-flights-aplenty/?hpt=hp_t2

Re: A new name for software failure : the glitch

[Nasser M. Abbasi]

On 8/4/2012 1:18 PM, francois_fabien@hotmail.com wrote:
> In finance, software failure can be very expensive very fast.
> http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
>

fyi;

AdaCore issued statement on this error:

August 07, 2012

http://eon.businesswire.com/news/eon/20120807006365/en/AdaCore/Knight-Capital-Group/high-frequency-trading

some quotes

"It's clear that Knight's software was deployed without adequate verification."

"What is needed is a change in the way that such critical software
is developed and deployed."

"the aviation industry has demonstrated that safe, reliable real-time
software is possible, practical, and necessary"

May be this is a good chance for Ada to get into financial software, which
is now dominated by weakly typed and much less robust languages.

--Nasser

Re: A new name for software failure : the glitch

[Georg Bauhaus]

On 10.08.12 14:42, Nasser M. Abbasi wrote:
> On 8/4/2012 1:18 PM, francois_fabien@hotmail.com wrote:
>> In finance, software failure can be very expensive very fast.
>> http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
>>
>>
> 
> fyi;
> 
> AdaCore issued statement on this error:
> 
> August 07, 2012
> 
> http://eon.businesswire.com/news/eon/20120807006365/en/AdaCore/Knight-Capital-Group/high-frequency-trading
> 
> 
> some quotes
> 
> "It's clear that Knight's software was deployed without adequate verification."
> 
> "What is needed is a change in the way that such critical software
> is developed and deployed."
> 
> "the aviation industry has demonstrated that safe, reliable real-time
> software is possible, practical, and necessary"
> 
> May be this is a good chance for Ada to get into financial software, which
> is now dominated by weakly typed and much less robust languages.

Tricky. Neither Java nor OCaml can be called weakly typed
or not robust. APL implementations do not count as not robust
either, AFAIK. And, as the article mentions, it is not even clear
yet whether *any* formal verification software could have
prevented the effect;
from what I know, it is more likely an algorithmic error that might have
to do with "<=" and ">", not so much with type systems, or with other
qualities of programming languages.

AdaCore uses this opportunity to point out offerings that
are related to reliability and verification, but does not
specifically mention Ada.

Saying "Ada would have prevented" might turn out to be rather silly in
this case.

Re: A new name for software failure : the glitch

[Nasser M. Abbasi]

On 8/10/2012 8:56 AM, Georg Bauhaus wrote:

>
> Tricky. Neither Java nor OCaml can be called weakly typed
> or not robust. APL implementations do not count as not robust
> either, AFAIK.

Thanks, but I was thinking of high frequency trading software. I read that
mostly C++ is mainly used there. This is real-time, hundreds of
transactions in one second type of software. Yes, Java is strongly
typed also. I do not know anything myself about OCaml and
APL (did not even know that APL is still around).

>
> AdaCore uses this opportunity to point out offerings that
> are related to reliability and verification, but does not
> specifically mention Ada.
>
> Saying "Ada would have prevented" might turn out to be rather silly in
> this case.
>

yes.

--Nasser
  

Re: A new name for software failure : the glitch

[Georg Bauhaus]

On 10.08.12 16:16, Nasser M. Abbasi wrote:
> On 8/10/2012 8:56 AM, Georg Bauhaus wrote:
> 
>>
>> Tricky. Neither Java nor OCaml can be called weakly typed
>> or not robust. APL implementations do not count as not robust
>> either, AFAIK.
> 
> Thanks, but I was thinking of high frequency trading software.
So was I.

> I read that
> mostly C++ is mainly used there.

Which layer? I'll venture a guess that e.g. the special
wiring between New York and Chicago will not include
any language's big run-time system.

According to Duncan Sands (AdaCore video), a french bank
is/was using Ada in some transaction layers.

> This is real-time, hundreds of
> transactions in one second type of software. Yes, Java is strongly
> typed also. I do not know anything myself about OCaml and
> APL (did not even know that APL is still around).

APL is so popular in financial business that Morgan Stanley
had allowed A+ to be made. Another dialect is specially made
for fast processing of series of data (q, earlier K),
and typically sold in the financial market. Yes, that's
probably not the I/O layer that effects trades.

Re: A new name for software failure : the glitch

[Yannick Duchêne (Hibou57)]

Le Fri, 10 Aug 2012 15:56:11 +0200, Georg Bauhaus  
<rm.dash-bauhaus@futureapps.de> a écrit:

> On 10.08.12 14:42, Nasser M. Abbasi wrote:
>> On 8/4/2012 1:18 PM, francois_fabien@hotmail.com wrote:
>>> In finance, software failure can be very expensive very fast.
>>> http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
>>>
>>>
>>
>> fyi;
>>
>> AdaCore issued statement on this error:
>>
>> August 07, 2012
>>
>> http://eon.businesswire.com/news/eon/20120807006365/en/AdaCore/Knight-Capital-Group/high-frequency-trading
>>
>>
>> some quotes
>>
>> "It's clear that Knight's software was deployed without adequate  
>> verification."
>>
>> "What is needed is a change in the way that such critical software
>> is developed and deployed."
>>
>> "the aviation industry has demonstrated that safe, reliable real-time
>> software is possible, practical, and necessary"
>>
>> May be this is a good chance for Ada to get into financial software,  
>> which
>> is now dominated by weakly typed and much less robust languages.
>
> Tricky. Neither Java nor OCaml can be called weakly typed
> or not robust. APL implementations do not count as not robust
> either, AFAIK. And, as the article mentions, it is not even clear
> yet whether *any* formal verification software could have
> prevented the effect;

So the reason resides in specifications?

> from what I know, it is more likely an algorithmic error that might have
> to do with "<=" and ">"

You told either too much or not enough :-D

-- 
“Syntactic sugar causes cancer of the semi-colons.” [1]
“Structured Programming supports the law of the excluded muddle.” [1]
[1]: Epigrams on Programming — Alan J. — P. Yale University

Re: A new name for software failure : the glitch

[Randy Brukardt]

"Nasser M. Abbasi" <nma@12000.org> wrote in message 
news:k02vjh$sd6$1@speranza.aioe.org...
> On 8/4/2012 1:18 PM, francois_fabien@hotmail.com wrote:
>> In finance, software failure can be very expensive very fast.
>> http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
>>
>
> fyi;
>
> AdaCore issued statement on this error:
>
> August 07, 2012
>
> http://eon.businesswire.com/news/eon/20120807006365/en/AdaCore/Knight-Capital-Group/high-frequency-trading

I thought it was interesting that barely 24 hours after issuing that 
statement, AdaCore's e-mail broke, causing all kinds of mess for people 
(like me) e-mailing them. Same sort of problem (probably didn't cost $440M, 
though). A reminder that Ada doesn't solve everything.

                                       Randy.

Re: A new name for software failure : the glitch

[Florian Weimer]

* Nasser M. Abbasi:

> On 8/4/2012 1:18 PM, francois_fabien@hotmail.com wrote:
>> In finance, software failure can be very expensive very fast.
>> http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
>>
>
> fyi;
>
> AdaCore issued statement on this error:
>
> August 07, 2012
>
> http://eon.businesswire.com/news/eon/20120807006365/en/AdaCore/Knight-Capital-Group/high-frequency-trading
>
> some quotes

| Last week an error in some automated high-frequency trading software
| from Knight Capital Group caused the program to go seriously amok,
| and when the cyberdust cleared, the company was left holding the
| bill for almost a half-billion dollars to cover the erroneous
| trades.

There are now reports that it wasn't a software error, that is, the
software worked as specified.

The quality of typical Ada marketing material is atrocious.

Re: A new name for software failure : the glitch

[jpwoodruff]

On Saturday, August 11, 2012 1:22:15 PM UTC-6, Florian Weimer wrote:

<...>

> There are now reports that it wasn't a software error, that is, the
> 
> software worked as specified.
> 
> 

Let me advance an hypothesis about the "glitch".  

We are told that the trading software executed numerous "buy" orders,
and the losses accrued when the company needed to unwind those positions.

Suppose that the database representing the software's state had become
corrupted, conceivably showing zero holdings, at the start of business
on the day.

The logic of the program might credibly buy lots of securities in
order to create the positions that the logic *correctly* determined
were required.

Then when the database is corrected to show the true positions, the
logic would demand quick dumping of the redundant purchases.

By this hypothesis the code logic is not flawed; a configuration
error cost 0.4e9 dollars.

John