From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,INVALID_MSGID autolearn=no autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,446231e9f9fb9a1c X-Google-Attributes: gid103376,public From: dewar@cs.nyu.edu (Robert Dewar) Subject: Re: ACVC tests Date: 1996/05/06 Message-ID: #1/1 X-Deja-AN: 153389504 references: <831410279.2370.2@assen.demon.co.uk> organization: Courant Institute of Mathematical Sciences newsgroups: comp.lang.ada Date: 1996-05-06T00:00:00+00:00 List-Id: John McCabe said, answering me ">extremely difficult to get ANY tecnical input from anyone. Even vendors >do not in general look at the tests in advance of the formal release >of the suite, and it is extremely rare to get any technical input from >users on the tests (I can't remember any example of such). Thus the >phiolosophy behind the committee was precisely to get at least *some* >users, implementors and testers looking at the test carefully in advance. I am surprised that the vendors attitude here since one would have thought they would be keen to be the first on the market with a validated product." Robert replies not surprising at all. The suite gets frozen several months before it is usable for testing, and that is when vendors really start to look at it. Looking at the suite earlier than this is not efficient, since it may change under you. As for rushing to be first, not necessarily, I don't think there are users out there who buy a compiler just because it is the first on the block to be validated, nor should they. Sure there is to some extent a race to be validated first, but it is not the most important criterion in choosing a validation schedule. Back to the issue of looking early. The trouble is that it is always better to have someone else smoke out the errors, since challenging the tests takes considerable time and energy. Thus the incentives are wrong. It is not obvious how to fix this, but as I say the committee was set up to make sure that at least there would be some formal input earlier. This was tried earlier (actually at my insistence that it would be useful) with ACVC 1.10. A committee was set up consisting of several government folks and me -- in practice I was the only one who showed up for meetings (you can see traces of that work, length_check and enum_check still have my name on somewhere :-) Anyway, this time, we have a reasonable sized functoining committee, which i think has been very helpful in improving the quality of the tests, but the more eyes the better, and the ACVC development team welcomes outside input.