Re: ACVC tests

[dewar@cs.nyu.edu (Robert Dewar)]

John McCabe said, answering me

">extremely difficult to get ANY tecnical input from anyone. Even vendors
>do not in general look at the tests in advance of the formal release
>of the suite, and it is extremely rare to get any technical input from
>users on the tests (I can't remember any example of such). Thus the
>phiolosophy behind the committee was precisely to get at least *some*
>users, implementors and testers looking at the test carefully in advance.

I am surprised that the vendors attitude here since one would have
thought they would be keen to be the first on the market with a
validated product."


Robert replies

not surprising at all. The suite gets frozen several months before it
is usable for testing, and that is when vendors really start to look
at it. Looking at the suite earlier than this is not efficient, since
it may change under you.

As for rushing to be first, not necessarily, I don't think there are
users out there who buy a compiler just because it is the first on the
block to be validated, nor should they. Sure there is to some extent
a race to be validated first, but it is not the most important 
criterion in choosing a validation schedule.

Back to the issue of looking early. The trouble is that it is always
better to have someone else smoke out the errors, since challenging
the tests takes considerable time and energy. Thus the incentives
are wrong. It is not obvious how to fix this, but as I say the
committee was set up to make sure that at least there would be
some formal input earlier.

This was tried earlier (actually at my insistence that it would be
useful) with ACVC 1.10. A committee was set up consisting of
several government folks and me -- in practice I was the only
one who showed up for meetings (you can see traces of that work,
length_check and enum_check still have my name on somewhere :-)

Anyway, this time, we have a reasonable sized functoining committee,
which i think has been very helpful in improving the quality of
the tests, but the more eyes the better, and the ACVC development
team welcomes outside input.