Re: ACATS on Wikipedia.

["Randy Brukardt" <randy@rrsoftware.com>]

"Martin Krischik" <krischik@users.sourceforge.net> wrote in message
news:1204160.MAhx02haZG@linux1.krischik.com...
> Hello again
>
> Anybody got an intelligent answer to this question:
>
> http://en.wikipedia.org/wiki/Talk:Ada_Conformity_Assessment_Test_Suite

I'm not sure what question you are referring to (there doesn't seem to be
one there):

>It is incorrect to say that ACATS ensures a compiler conforms to a language
standard.
>When BSI were in the compiler validation business the validations
certificates they issued
>said "This is to certify that the language processor identified below was
found to contain
>no errors when tested with the identified validation suite." No mention of
conforming to a
>language standard there. I suspect the Ada validation certificates say
something similar
>(I don't have one to hand).

Recent ACATS validation reports are available on line. (None are that
recent, unfortunately.) I forget the exact contents of the certificate, but
the preface to the test report includes the following:

Conformity assessment does not ensure that a processor has no
nonconformities to the Ada standard other than those, if any, documented in
this report.  The compiler vendor declares that the tested processor
contains no deliberate deviation from the Ada standard; a copy of this
Declaration of Conformity is presented immediately after the certificate.


Also relevant is the second paragraph of the background of the current ACAA
procedures (http://www.adaic.com/compilers/procs/3.0/ACAP30.html):

It is important to note the scope and intent of conformity assessment. The
purpose of conformity assessment is to ensure that Ada processors achieve a
high degree of conformity with the Ada standard ([Ada95] as corrected by
[TC1]). Characteristics such as performance and suitability for a particular
application are not specified by the standard, and thus are outside the
scope of Ada conformity assessment. Moreover, the ACATS is a set of test
programs intended to check broadly for correct implementation; it is not
possible to exhaustively test for conformity. Thus, conformity is checked
only to the extent of these tests; processors that are certified as
conforming may fail to conform to the standard in ways peculiar to each,
under particular circumstances.

----

The other statement in the original is:

>How do we know that a validation suite correctly implements the
requirements contained in a language >standard? One answer is here

The short answer is that we don't. Indeed, there have been a handful of
cases where we've changed the Standard to match the ACATS tests, because
implementations have been passing the tests for years, and strictly
following the wording of the standard would have been incompatible with
actual practice.

The longer answer is that we do via a number of ways:

1) Vetting of test objectives (the test objective must be reasonably clear,
and clearly relate to one or more rules in the Standard) [the ARG now has
this responsibility];

2) Dispute procedures (where vendors and users can object to a test that
they think is wrong - this prevents incorrect tests from lasting a long
time);

3) Tracking of coverage (so that we can tell which rules in the standard
have been tested, and which ones have not been - which guide which tests are
the highest priority for construction).

There are a number of articles about this on adaic.org (look under Compilers
and Conformity).

Feel free to use whatever part of this you want.



                     Randy Brukardt, ACAA Technical Agent.