Re: Killing software and certification

["Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de>]

On 28/03/2018 15:54, Alejandro R. Mosteo wrote:
> On 27/03/18 21:25, Dmitry A. Kazakov wrote:
>> On 2018-03-27 20:32, Alejandro R. Mosteo wrote:

>>> I'm not in the industry, and I'd be surprised that unverified 
>>> software were allowed to run in civilian environments where failures 
>>> basically amount to a very dangerous situation.
>>
>> Why should it surprise you? How are you going to verify it? Black box 
>> test is impossible. White box test isn't either, assuming any NN 
>> involved. There is nothing to prove.
> 
> I can think of an spectrum of regulatory/practical positions between 
> 'nothing can be done, so everyone brace' and 'this won't ever fail, 
> under every [un]conceivable situation'. It's the apparent nonchalance of 
> the general public that coexists with these testing cars, the 
> brashness/recklessness of those expecting to get rich with it and the 
> apparent willingness of politicians that I find fascinating (that's the 
> first ones that comes to mind).
> 
> I can understand the appeal for politicians to be the first city with a 
> working fleet (or whatever contributions they're getting to favor live 
> testing). As a technophile, I want autonomous cars to become reality, so 
> I can understand that too. As a researcher familiar with the algorithms 
> involved and with the kind of C/C++/Python heaps that implement them I 
> get chills about thinking that a car can be on the highway with a 
> semi-awake safety driver as the only fallback in a split second.

As much as I hate C and the ilk, the problem is much deeper, I'm afraid. 
Even if it were 100% SPARK Ada, a self teaching system [I don't consider 
here simple cases where one can prove convergence of the training method 
to a defined goal], is in the same relation as a program is to the CPU.

You can verify and certify the CPU as much as you wish. That would say 
nothing about the program running on it. The weights of a NN is a 
"program". The system core written in whatever language is a "CPU".

I have no slightest idea of an approach to define correctness of the 
trained weights, even less about proving it. It is a fundamental 
challenge we will have to deal with, if this kind of "emerging 
programming" to take hold.

A kind of biblical disaster when you, like God did, see in dismay what 
your creatures would do after you granted them "free will". Would you 
drown them all? (:-))

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de