Re: ACVC tests

[heatwole@erols.com (Kevin D. Heatwole)]

In article <831410279.2370.2@assen.demon.co.uk>, john@assen.demon.co.uk
(John McCabe) wrote:

>dewar@cs.nyu.edu (Robert Dewar) wrote:
>
>>Of course the team has not "contacted most users", who would number in
>>the tens of thousands. In fact the experience in the past has been that,
>>although the ACVC tests were generally available for review, it has been
>>extremely difficult to get ANY tecnical input from anyone. Even vendors
>>do not in general look at the tests in advance of the formal release
>>of the suite, and it is extremely rare to get any technical input from
>>users on the tests (I can't remember any example of such). Thus the
>>phiolosophy behind the committee was precisely to get at least *some*
>>users, implementors and testers looking at the test carefully in advance.
>
>I am surprised that the vendors attitude here since one would have
>thought they would be keen to be the first on the market with a
>validated product.
>
>Perhaps this attitude should be considered further. It may be, for
>example, that they consider validation as a hinderance that does not
>add any real value to their product. It could then be suggested that
>this attitude is promoted by the users (like myself, or the mandators
>like Dornier) not being strict enough with the vendors and asking for
>VSRs and ACES results etc.
>
>Any comments?

Well, I work for OC Systems and we would be considered an Ada vendor.  I
will not pretend to speak for OC Systems, but I can offer my attitude on
this subject.

The ACVC tests are a great resource for the compiler vendors and provide
a measure of reassurance to our customers that we do indeed implement
the Ada language.  The test suite has literally cost millions of dollars to
develop and is probably the best such suite of tests that I have ever
encountered.  If this test suite did not exist, a individual compiler vendor
would be forced to develop such a test suite internally.  This burden would
increase the cost of Ada compilers to our end users (Ada compilers are
already an incredibly expensive piece of software to develop and we need
all the help we can get to cut these costs).

Also as a vendor, we do not consider validation "as a hinderance".
Rather, it is
an essential part of developing a commercially acceptable Ada compiler.
Indeed, even if there was no requirement to "validate" using this test suite,
we would still want to incorporate this set of tests into our test suite.
Vendors simply do not want to delivery buggy product (it will hurt our
bottom line eventually).

I wish that OC Systems had the resources to participate in the development
of the new Ada95 ACVCs more.  Unfortunately, developing an Ada95 compiler
is a huge undertaking and resources must be allocated to this effort in
view of market realities and current cash flow.  We have chosen to try to
implement the new features in Ada95 first (trying to implement them as
best we can independent of running the new ACVCs - which are still in
development and still have some bugs in them).  When the compiler reaches
a level of maturity that we feel it is beneficial to start running/debugging
the ACVC tests, we focus our efforts in this direction.  In this regards, the
ACVCs provide valuable feedback to vendors developing Ada95 compilers as
to how close the compiler is to being completed/robust.  If you fail most
of the ACVCs the first time around, you probably didn't do a good job
implementing the features.  Anyway, this is valuable feedback.

As to the current ACVC 2.1 team (led by SAIC), they are doing an exellent job.
We monitor all the email that they exchange and have some incite into
their process.  They have some very good people on the project.  They are
tasked to develop a test suite before there are compilers that implement the
rules they are testing.  This is a very difficult thing to do, since some of the
rules in Ada are very subtle and without a reliable compiler to tell you that
you messed up, it is very big undertaking (by the way, the ACVC team does have
access to all the various vendors' Ada95 compilers which are at different
levels of completeness.  The ACVC team does a great job of sending the vendors
bug reports when they encounter problems.).

>
>In general, however, I applaud your attempts here to extract more
>detailed comments from users on the new test suite and, when I have
>time, I'll see if I can be of any help!

I think you had best hurry here if you expect to be of any help for ACVC 2.1.
This test suite is nearing completion and you better get your licks in now.
They have been developing the new Ada 95 tests for over two years now (I
am not really sure how long they have been at it, but there are new tests
from 1994 - the Ada83 tests in the suite were developed over the a number
of years in the 1980s).

Kevin Heatwole
OC Systems, Inc.