From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable autolearn_force=no version=3.4.4 Path: eternal-september.org!reader01.eternal-september.org!reader02.eternal-september.org!feeder.eternal-september.org!nntp-feed.chiark.greenend.org.uk!ewrotcd!newsfeed.xs3.de!io.xs3.de!news.jacob-sparre.dk!franka.jacob-sparre.dk!pnx.dk!.POSTED.rrsoftware.com!not-for-mail From: "Randy Brukardt" Newsgroups: comp.lang.ada Subject: Re: Five Years After Date: Thu, 25 Jan 2018 22:28:30 -0600 Organization: JSA Research & Innovation Message-ID: References: <942235344.537649945.074508.laguest-archeia.com@nntp.aioe.org> <292710c7-3520-406f-9e6b-1fb030ada211@googlegroups.com> Injection-Date: Fri, 26 Jan 2018 04:28:30 -0000 (UTC) Injection-Info: franka.jacob-sparre.dk; posting-host="rrsoftware.com:24.196.82.226"; logging-data="10398"; mail-complaints-to="news@jacob-sparre.dk" X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.5931 X-RFC2646: Format=Flowed; Original X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.7246 Xref: reader02.eternal-september.org comp.lang.ada:50154 Date: 2018-01-25T22:28:30-06:00 List-Id: "Maciej Sobczak" wrote in message news:e7a16703-b24e-42aa-81fd-7026932d6c32@googlegroups.com... >> Everyone agrees that the ACATS test suite offers a lot of value to both >> implementers and users (implementers because an independently sourced and >> trustworthy set of tests is very helpful to creating and maintaining an >> Ada >> compiler -- it is especially good IMHO in preventing errors of omission, >> which are often the hardest to find; > >Sorry, but I cannot agree here. There is nothing that saves ACATS authors >from errors of omission. Of course there is -- extensive coverage documentation. Of course that's not going to find *every* possible omission, but it tries to check that every sentence in the Standard is tested at least once. And please note that "valuable" is not the same as "the only thing anyone ever needs"!! The ACATS is not now, and never was, intended to find compiler bugs per se. It is intended to prove a level of compliance with the Ada Standard - for instance that a task terminates when it is supposed to, and not sooner or later. It can't try to check every combination of features - such that a task given in a block in a generic subprogram that has as discriminant that is a generic formal parameter terminates on time -- that would be an impossible task. > The language is itself so complex that creating a complete (and I really > mean > formally complete) set of requirements is impractical and without such a > set it > is not possible to create a complete (again, I mean it) set of tests. Nobody is trying to do this, because it would be impossible (practically) even if one had unlimited money. Real compiler testing has to be a combination of black box tests (the ACATS test suite being a great example) and tests designed to ensure code coverage and other sorts of white box tests. And I think every vendor also has a bunch tests created from user bug reports, mainly to try to ensure regressions don't happen. But there is no single technique that could provide anything like perfection for any real programming language. (Maybe for toy subsets.) > As a result, there are areas of the language that were quit well > understood > by language designers and more or less well understood by language users, > that are not covered by ACATS tests at all and therefore not properly > implemented by compilers. I have myself found several compiler bugs by > only > playing with and shuffling ideas from textbook examples (and we have > discussed them here on this group). Repeat: The ACATS is not intended to find compiler bugs, only gross mistakes in the implementation of Ada features. (The obvious example is not implementing something at all.) Besides, the ACATS only covers about 45% of the first level objectives, based on the parts of the Standard that I've analyzed to date. (Coverage might be a bit better overall, as some of the very basic sections haven't been analyzed yet, like most of the statements in Chapter 5 and expressions in Chapter 4.) (A lot of those aren't very interesting or likely, but untested is always a possibility for an omission or gross error.) There isn't ever going to be enough funding to close that gap. >> This certainly is true since I'm paid money every year to maintain and >> enhance the test suite > > Ironically, the bare fact that you are still being paid for this work is > itself a proof > that these tests are still not complete. ;-) ;-) ;-) Completeness is a goal that could never really be reached (probably some part of the objectives would be untestable in practice). And it doesn't matter, the test suite is still *valuable* even if not *perfect*. >But these facts do not always translate into satisfactory user experience. The ACATS has absolutely nothing to do with the user experience. ... > In which case a different approach to compiler validation might be needed. It will never be the least bit hard to find a bug in a compiler (for any realistic language). Surely not in an Ada compiler, C++ compiler, Java compiler, and so on. At least not until humans are not allowed near computers. ;-) Randy.