ACATS on Wikipedia.

ACATS on Wikipedia.

[Martin Krischik]

Hello again

Anybody got an intelligent answer to this question:

http://en.wikipedia.org/wiki/Talk:Ada_Conformity_Assessment_Test_Suite

Martin
-- 
mailto://krischik@users.sourceforge.net
Ada programming at: http://ada.krischik.com

Re: ACATS on Wikipedia.

[Randy Brukardt]

"Martin Krischik" <krischik@users.sourceforge.net> wrote in message
news:1204160.MAhx02haZG@linux1.krischik.com...
> Hello again
>
> Anybody got an intelligent answer to this question:
>
> http://en.wikipedia.org/wiki/Talk:Ada_Conformity_Assessment_Test_Suite

I'm not sure what question you are referring to (there doesn't seem to be
one there):

>It is incorrect to say that ACATS ensures a compiler conforms to a language
standard.
>When BSI were in the compiler validation business the validations
certificates they issued
>said "This is to certify that the language processor identified below was
found to contain
>no errors when tested with the identified validation suite." No mention of
conforming to a
>language standard there. I suspect the Ada validation certificates say
something similar
>(I don't have one to hand).

Recent ACATS validation reports are available on line. (None are that
recent, unfortunately.) I forget the exact contents of the certificate, but
the preface to the test report includes the following:

Conformity assessment does not ensure that a processor has no
nonconformities to the Ada standard other than those, if any, documented in
this report.  The compiler vendor declares that the tested processor
contains no deliberate deviation from the Ada standard; a copy of this
Declaration of Conformity is presented immediately after the certificate.


Also relevant is the second paragraph of the background of the current ACAA
procedures (http://www.adaic.com/compilers/procs/3.0/ACAP30.html):

It is important to note the scope and intent of conformity assessment. The
purpose of conformity assessment is to ensure that Ada processors achieve a
high degree of conformity with the Ada standard ([Ada95] as corrected by
[TC1]). Characteristics such as performance and suitability for a particular
application are not specified by the standard, and thus are outside the
scope of Ada conformity assessment. Moreover, the ACATS is a set of test
programs intended to check broadly for correct implementation; it is not
possible to exhaustively test for conformity. Thus, conformity is checked
only to the extent of these tests; processors that are certified as
conforming may fail to conform to the standard in ways peculiar to each,
under particular circumstances.

----

The other statement in the original is:

>How do we know that a validation suite correctly implements the
requirements contained in a language >standard? One answer is here

The short answer is that we don't. Indeed, there have been a handful of
cases where we've changed the Standard to match the ACATS tests, because
implementations have been passing the tests for years, and strictly
following the wording of the standard would have been incompatible with
actual practice.

The longer answer is that we do via a number of ways:

1) Vetting of test objectives (the test objective must be reasonably clear,
and clearly relate to one or more rules in the Standard) [the ARG now has
this responsibility];

2) Dispute procedures (where vendors and users can object to a test that
they think is wrong - this prevents incorrect tests from lasting a long
time);

3) Tracking of coverage (so that we can tell which rules in the standard
have been tested, and which ones have not been - which guide which tests are
the highest priority for construction).

There are a number of articles about this on adaic.org (look under Compilers
and Conformity).

Feel free to use whatever part of this you want.



                     Randy Brukardt, ACAA Technical Agent.

Re: ACATS on Wikipedia.

[Derek M. Jones]

Randy,

> I'm not sure what question you are referring to (there doesn't seem to be
> one there):

The reference was to the talk page of the following Wikipedia entry:

http://en.wikipedia.org/wiki/ACATS

which included the sentence (now changed):

"The ACATS test suite is a very important aspect of the Ada programming 
language as it ensures that all major Ada compilers conform to the 
language standard."

to include:

> Conformity assessment does not ensure that a processor has no
> nonconformities to the Ada standard other than those, if any, documented in
> this report.  The compiler vendor declares that the tested processor
> contains no deliberate deviation from the Ada standard; a copy of this
> Declaration of Conformity is presented immediately after the certificate.

and:

> Also relevant is the second paragraph of the background of the current ACAA
> procedures (http://www.adaic.com/compilers/procs/3.0/ACAP30.html):
> 
> It is important to note the scope and intent of conformity assessment. The
> purpose of conformity assessment is to ensure that Ada processors achieve a
> high degree of conformity with the Ada standard ([Ada95] as corrected by
> [TC1]). Characteristics such as performance and suitability for a particular
> application are not specified by the standard, and thus are outside the
> scope of Ada conformity assessment. Moreover, the ACATS is a set of test
> programs intended to check broadly for correct implementation; it is not
> possible to exhaustively test for conformity. Thus, conformity is checked
> only to the extent of these tests; processors that are certified as
> conforming may fail to conform to the standard in ways peculiar to each,
> under particular circumstances.

I did not want to include the following because I was not sure
you would feel it appropriate in a Wikipedia entry.

>> How do we know that a validation suite correctly implements the
> requirements contained in a language >standard? One answer is here
> 
> The short answer is that we don't. Indeed, there have been a handful of
> cases where we've changed the Standard to match the ACATS tests, because
> implementations have been passing the tests for years, and strictly
> following the wording of the standard would have been incompatible with
> actual practice.
> 
> The longer answer is that we do via a number of ways:
> 
> 1) Vetting of test objectives (the test objective must be reasonably clear,
> and clearly relate to one or more rules in the Standard) [the ARG now has
> this responsibility];
> 
> 2) Dispute procedures (where vendors and users can object to a test that
> they think is wrong - this prevents incorrect tests from lasting a long
> time);
> 
> 3) Tracking of coverage (so that we can tell which rules in the standard
> have been tested, and which ones have not been - which guide which tests are
> the highest priority for construction).
> 
> There are a number of articles about this on adaic.org (look under Compilers
> and Conformity).

Wikipedia uses a creative commons license.  So I'm pleased you said:

> Feel free to use whatever part of this you want.