From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM autolearn=unavailable autolearn_force=no version=3.4.4 X-Received: by 10.55.154.9 with SMTP id c9mr7651678qke.33.1516914566679; Thu, 25 Jan 2018 13:09:26 -0800 (PST) X-Received: by 10.157.112.213 with SMTP id w21mr119794otj.2.1516914566494; Thu, 25 Jan 2018 13:09:26 -0800 (PST) Path: eternal-september.org!reader01.eternal-september.org!reader02.eternal-september.org!feeder.eternal-september.org!news.unit0.net!peer01.am4!peer.am4.highwinds-media.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!s47no915391qta.0!news-out.google.com!e39ni56qtk.1!nntp.google.com!i22no917826qta.1!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail Newsgroups: comp.lang.ada Date: Thu, 25 Jan 2018 13:09:26 -0800 (PST) In-Reply-To: Complaints-To: groups-abuse@google.com Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=188.146.99.114; posting-account=bMuEOQoAAACUUr_ghL3RBIi5neBZ5w_S NNTP-Posting-Host: 188.146.99.114 References: <942235344.537649945.074508.laguest-archeia.com@nntp.aioe.org> <292710c7-3520-406f-9e6b-1fb030ada211@googlegroups.com> User-Agent: G2/1.0 MIME-Version: 1.0 Message-ID: Subject: Re: Five Years After From: Maciej Sobczak Injection-Date: Thu, 25 Jan 2018 21:09:26 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Received-Body-CRC: 2060231638 X-Received-Bytes: 3882 Xref: reader02.eternal-september.org comp.lang.ada:50151 Date: 2018-01-25T13:09:26-08:00 List-Id: > Everyone agrees that the ACATS test suite offers a lot of value to both= =20 > implementers and users (implementers because an independently sourced and= =20 > trustworthy set of tests is very helpful to creating and maintaining an A= da=20 > compiler -- it is especially good IMHO in preventing errors of omission,= =20 > which are often the hardest to find; Sorry, but I cannot agree here. There is nothing that saves ACATS authors f= rom errors of omission. The language is itself so complex that creating a c= omplete (and I really mean formally complete) set of requirements is imprac= tical and without such a set it is not possible to create a complete (again= , I mean it) set of tests. As a result, there are areas of the language tha= t were quit well understood by language designers and more or less well und= erstood by language users, that are not covered by ACATS tests at all and t= herefore not properly implemented by compilers. I have myself found several= compiler bugs by only playing with and shuffling ideas from textbook examp= les (and we have discussed them here on this group). > This certainly is=20 > true since I'm paid money every year to maintain and enhance the test=20 > suite Ironically, the bare fact that you are still being paid for this work is it= self a proof that these tests are still not complete. ;-) ;-) ;-) > Ada compilers accept essentially the same language=20 > since they've all been tested with the same set of tests). Unfortunately, no. Every single version of whatever compiler accepts a diff= erent language than its previous version. This is how the bugs are fixed (o= r introduced), among others. The problem is - non of our programs is really a mashup of unit tests from = ACATS, the state space is too big for this. It is really good that the test= s run fine. Heck - it's really good that they exist at all! But these facts= do not always translate into satisfactory user experience. Ultimately, my = intent is to compile my own programs, not the tests - and apparently, the w= hole concept of compiler testing is too big to extrapolate from a set of un= it tests to a set of all possible user programs. Maybe Ada as a language ha= s already crossed the border at which such an approach is valid at all. In which case a different approach to compiler validation might be needed. --=20 Maciej Sobczak * http://www.inspirel.com