Re: Compiler quality

["Randy Brukardt" <randy@rrsoftware.com>]

"Georg Bauhaus" <rm.tsoh.plus-bug.bauhaus@maps.futureapps.de> wrote in 
message news:492fc7bc$0$31869$9b4e6d93@newsspool3.arcor-online.net...
> Martin wrote:
>
>> To be fair to anon...that's not a SYNTAX bug. His claim was that
>> compilers should be able to be 100% tested for syntax bugs by limiting
>> the input so much that's it becomes practical to do on current
>> machines.
>
> Considering the Ada 83 rule that 'Base must be prefixed to
> 'Another_Attribute (i.e., have context), do Ada compilers
> typically  check this rule in the parts handling syntax?
> (In the case of GNAT, the sem*.ad? files refer to the 'Base
> attribute, AFAICS.)

I don't think so; the typical syntax for attributes is something like:

    prefix APOSTROPHE IDENTIFIER

where the capitalized items are terminals ' and an identifier. The only 
"identifier"s that have special handling in the syntax are the reserved 
words, and "base" is not reserved.

A data point about anon's more general point. We discovered a syntax 
processing error (in our case, an error in the grammar input to the table 
generator for the syntax part of Janus/Ada 95) more than 6 1/2 years after 
the Ada 95 grammar was completed (and the associated compiler was in use by 
many people during the time period of February 1996 to November 2002). The 
error was allowing the declaration of abstract subprograms in places that 
they are not allowed, such as generic formal parameters.

The point is that no amount of (sane) testing can find unintended additional 
capabilities -- because the only way to do that is to guess the possible 
errors, and that is not a practical thing to do as there is a very low 
probability of actually finding any errors. Thus such testing is not a good 
use of limited testing effort; there are many more things that are better 
uses of effort. For this reason, the ACATS does not test separately for 
syntax errors unless there is a significant probability that the error will 
be detected outside of the syntax portion of the compiler. After all, most 
compiler parsers are generated by tools these days, so the errors will 
almost certainly be limited to simple errors (omission, transposition, or 
substitution) in the input to those tools and can be detected by pretty much 
any use of a particular feature of the language. Thus the tests for other 
language rules provide enough testing for the syntax proper. (I believe that 
GNAT uses a hand-coded parser, and thus is at somewhat more risk for syntax 
processing errors than most other compilers, but that still seems to be a 
low priority error to detect compared to violations of other language 
rules.)

Testing in general is a poor way to eliminate errors in a compiler. Back in 
1997, I ran some coverage tests to determine what code in Janus/Ada has been 
executed in tests. (That doesn't prove that such code will work in all 
circumstances, but code that is executed is more likely to be correct than 
code that has never been known to be executed.) I ran pretty much every test 
and program that we had access to at the time (more than a thousand in-house 
tests, plus the ACATS, plus Claw and its tests, plus the compiler source 
code), and still was able only to show coverage on about 70% of the code in 
the part of the compiler I was testing. While some of the unreached paths 
detected internal compiler errors (that is, represented redundant checks), 
many of them represented combinations of language features that had simply 
not been tested. (Looking at a randomly selected piece of the compiler, some 
such cases are a timed internal protected entry call; selection of a 
component from a formal parameter of an unconstrained record type; and 
selection of a component using the "current instance" of a protected 
object.) My original idea was to write some test cases to try to reach the 
untouched code, but the magnitude of the task made it impractical. (It would 
make more sense to use "white box" unit tests to force coverage, but that 
brings up the possibility of spending a lot of time testing things that 
can't happen in real input programs. Not a great choice either way.)

                                        Randy.