Re: Killing software and certification

["Alejandro R. Mosteo" <alejandro@mosteo.com>]

On 27/03/18 21:25, Dmitry A. Kazakov wrote:
> On 2018-03-27 20:32, Alejandro R. Mosteo wrote:
>> On 23/03/18 10:05, Jeffrey R. Carter wrote:
>>
>>> Autopilots have to be certified to DO178B/C. They'll continue to be 
>>> written in Ada and not kill us.
>>>
>>> Self-driving cars, though operating in a much more complex 
>>> environment, don't seem to need any certification, and will probably 
>>> kill us all.
>>
>> I'd like to revisit this point in light of the recent Uber news, but 
>> also let's not forget for example this one which is simpler than fully 
>> autonomous cars:
>>
>> https://en.wikipedia.org/wiki/2009–11_Toyota_vehicle_recalls
>>
>> I'm not in the industry, and I'd be surprised that unverified software 
>> were allowed to run in civilian environments where failures basically 
>> amount to a very dangerous situation.
> 
> Why should it surprise you? How are you going to verify it? Black box 
> test is impossible. White box test isn't either, assuming any NN 
> involved. There is nothing to prove.

I can think of an spectrum of regulatory/practical positions between 
'nothing can be done, so everyone brace' and 'this won't ever fail, under 
every [un]conceivable situation'. It's the apparent nonchalance of the 
general public that coexists with these testing cars, the 
brashness/recklessness of those expecting to get rich with it and the 
apparent willingness of politicians that I find fascinating (that's the 
first ones that comes to mind).

I can understand the appeal for politicians to be the first city with a 
working fleet (or whatever contributions they're getting to favor live 
testing). As a technophile, I want autonomous cars to become reality, so 
I can understand that too. As a researcher familiar with the algorithms 
involved and with the kind of C/C++/Python heaps that implement them I 
get chills about thinking that a car can be on the highway with a 
semi-awake safety driver as the only fallback in a split second.

All in all, I find the tension between all forces in conflict captivating 
to watch from a distance.

> 
>> After a bit of googling around I see that there are automotive 
>> standards for certification (the one I see more often mentioned is ISO 
>> 26262). About enforcement, I also read that regulation varies by US 
>> state. I haven't found anything definite about Europe.
> 
> If any certification will ever be set up, it will be certification of the 
> tools and developing processes/teams, not certification of the actual 
> software. That is the usual backdoor to go around any questions about 
> correctness.

I understand that now.

> 
>> Also, it's not the same software for a drive-by-wire part than for an 
>> autonomous car.
>>
>> I'm under the impression that these autonomous car outfits are at the 
>> time closer to a research environment than to that of a 
>> well-established industry. I.e., code is produced faster, hence bugs 
>> are more likely.
> 
> The code used in the ECU and other car subsystems is not any better from 
> that point of view. It is much simpler, deploys well-established 
> algorithms and, importantly, is testable with a large set of test 
> hardware and software available. That is the reason why it works better. 
> But otherwise, it is just same. There is no any guarantee for it to work.
> 
>> In the end I'm not sure where I want to go with this post. It's simply 
>> that I find the topic very interesting. If anyone with actual knowledge 
>> on the status of automotive software certification (or any informed 
>> ideas) would share some thoughts I'll be eager to read.
> 
> My understanding is that it is possible to certify about anything 
> regardless the correctness.

That's something to put in a frame :-)