Re: Teaching new tricks to an old dog (C++ -->Ada)

["Randy Brukardt" <randy@rrsoftware.com>]

"Jerry Coffin" <jcoffin@taeus.com> wrote in message
news:1111015648.669832.140620@o13g2000cwo.googlegroups.com...
...
> So, the question is not one of whether you exist -- but of whether you
> and/or your testing should be trusted. In my view, your own posts have
> thrown this into considerable doubt (at best).

The reason Jerry comes to this conclusion is based on a single summary
statement from a 5 year-old summary article; he never looks at the testing
procedures or the ISO standard that governs those procedures. Moreover, he
seems to claim reasons for this conclusion that are clearly stated in the
article (at least I hope they're clear; I'm not that great a writer...).

I'm always amazed at the lengths that some opponents of Ada will go in order
to discredit any advantage of the language. Is this because they feel
threatened by the language? (I can't imagine why.) Or they have a deep
seated hatred of things that start with 'A'? I don't know. You can't please
(or even reason with) all of the people all of the time.

Anyway, for the benefit.of anyone else still reading this thread, I'll
explain how the testing is done, and why it can be trusted. (Whether such
testing has value in any case is another issue that could be argued forever,
but that's not the question.)

The testing process is an instantiation of the ISO testing standard (ISO/IEC
18009). The detailed procedures that are followed are available on the AdaIC
website, along with the test suite itself. They were developed from the
preexisting AJPO procedures with the oversight of a panel of users,
laboratories, and vendors.

Actual testing is done by testing laboratories (ACALs). These have to be
independent of any Ada vendor. Vendors contract directly with the
laboratories for testing; this part of the process is unchanged from when
AJPO and NIST ran it.

The ACAA is the oversight agency. We provide maintenance to the test suite,
and judge disputes between the vendors and the testing labs. We also spot
check the laboratories work to insure that they have followed the
procedures. This is the role that previously was handled by AJPO (in an
office called the AVO).

The only real difference (other than the ISO standard, which codified the
rules that AJPO had followed) is that the ACAA is funded in part by the ARA.
The ACAA is primarily funded by certificate fees, but the ARA does make up
any shortfall. The ARA is an association of Ada product vendors (not just
compiler vendors).

While there is a potential conflict of interest, it is irrelevant in
practice for three reasons:
(1) The ACAA is only a judge. Moreover, technical issues are resolved by
discussion with a panel of Ada experts; if that is not satisfactory, the
ruling can be appealed to the ARG (the Ada standard maintenance group within
the Ada working group of ISO/IEC SC22 WG9). At most, the ACAA can influence
this process, not decide it.
  (2) ARA includes most of the major Ada vendors. It would be hard to have
an conformity issue that they all would agree on that would not also be in
the interest of all Ada vendors.
  (3) The testing in question is conformity assessment. It is not about
testing usability in any way, so it has little to do with the factors on
which vendors compete. The testing is designed to provide a yes/no answer
(well, it's a little more nuanced than that) as to whether a compiler is
compliant. There isn't a lot of leeway in this process for outside
influence.

While I could imagine an outcome from this process that would favor Ada
vendors over Ada users, that would seem to be counter-productive (why hurt
your customers?). Moreover, as the testing is about conformity and not
usability, most issues that matter to users don't even fall under the
testing umbrella. Of course, the main question (Does this compiler properly
implement the Ada Standard?) is of significant interest to users,
particularly those that will need to use more than one compiler over the
life of their project.

As to whether someone could cheat in the process, I would say it would not
be the least bit hard. The results of the tests are always handled by the
vendors before the testers see them, so there is always the possibility of
games. There is nothing new about this possibility (I know or suspect
several cases where results were falsied under the AJPO testing). I think it
actually would be harder now, because (1) I've been on the other side of the
fence, so I have some idea of what to look for; and (2) we require more
information in the test report than AJPO did. We also post the test reports
publically, so anyone can check on them. (The only AJPO test reports I ever
saw, before I was sent several cartons of old AVO material, were the ones we
had done.) We even have a public test results dispute procedure (which
thankfully never has been used), its unclear if AJPO even had one.

In any event, the real benefit is that there is a *single* conformity test
suite that *all* Ada vendors use for testing. That means that there can be
no confusion about what has and has not been tested. Most differences
between Ada compilers (beyond those allowed by the Standard) are those that
come from untested combinations of Ada features.

                               Randy Brukardt