Re: Ada Core Technologies and Ada95 Standards

[eachus@spectre.mitre.org (Robert I. Eachus)]

In article <3160EFBF.BF9@lfwc.lockheed.com> Ken Garlington <garlingtonke@lfwc.lockheed.com> writes:

  > In the ACVC 1.0 sources I received, each test had an identifier. I
  > did not receive a cross-reference of that identifier to the
  > requirements, as I recall.

   The naming of ACVC 1.x tests was such that they could be directly
traced to the requirements.  I don't remember a reverse mapping ever
being published, but I remember spending a couple hours with the
Implementors Guide, a list of the tests in ACVC 1.4 and Multics Emacs
to create one.  For DPS-6 Ada, we used it to trace failed tests to
requirements--it was often easier than figuring out what the test was
supposed to be testing--and for both DPS-6 Ada and Ada/SIL we kept a
copy with annotations as to non-applicable tests or edited and why the
tests were ignored or modified.

   (I have to admit that a LOT of the comments in the documents were
of the form "STUPID TEST: ..."  For example one test had two programs,
one of which opened a temporary file and attempted to write it's name
to another file.  The other program tried to open the temporary file,
using that name, and read the contents.  It took a lot of back and
forth to agree that the "right" behavior on our system was to raise
USE_ERROR for NAME of a temporary file, but most of that was over
NAME_ERROR vs USE_ERROR.  Of course the test did not anticipate the
possibility that the call to NAME would fail.)

  > Since I expect to be one of the end users, my being unfamiliar
  > with the process would tend to support my statement, wouldn't it?

   You are planning to validate a compiler?  (This is not a sarcastic
question.  We often put compilers through the ACVC process here
because there are potentially significant differences between the
system as validated and the configuration being used on a project.
Such an informal validation takes a few man-days, including
comparisons to the original VSR.)

  > 1. Then, perhaps, Mr. McCabe and I want something other than a
  > general validation facility. How about standards that each vendor
  > must meet for development and test -- SEI, ISO, and/or something
  > else?

   We also have several collections of domain specific tests we often
run.   If your project is going to make heavy use of generics or
tasking, or you are concerned with how the run-time interacts with
Unix signals, much better to find out up front if the compiler suits
your needs on your project up front.  A centralized validation
facility can never do that for you, they just set a reasonable base
level. 

  Robert Dewar said:

  > > It would not be practical to incorporate all test progrms for all bugs
  > > found in all compilers into the ACVC (it would rapidly have tens of
  > > thousands of tests, and become completely unmanagable).

  Very true, but some regression tests have made it into the ACVC
suite because they were indicative of more general problems.  In any
case, every decent compiler vendor maintains a suite of regression
tests, and runs it regularly.  The only cost--if you never find any
bugs--is the disk space and the weekend computer cycles.  But you do
find bugs, and often for the reason that gives the suite its name.
(Usually this happens when a module is rewritten from scratch because
it contains too many bug fixes.  Some of the fixes get lost in the
rewrite. In some cases this happens becuase the original fix got
buried under other fixes--remember why you rewrote this.)

  > Also, if the regression suite is that big, is that a good or a bad
  > thing? 

   Neither, it is a symptom of a mature compiler.  Some of the tests
come from actual bug fixes, many more are unit tests written to verify
additional features at each build.  Again, there is no reason not to
run a test again and again, once you have gone to the trouble of
writing (and verifying) it.

   > > Also, the effort of taking every
   > > bug and putting it into ACVC form is out of the question.

   > Again, a difference in philosophy: In my domain, _not_ having a
   > regression test for every bug found is out of the question.

   The AVO tries to put in regression tests for every bug in the ACVC
suite.  (And in some cases the ARG slaps them down.  If an ACVC test
tests for something harmful, then no test is the right position.  The
ARG created the category Pathology just for cases like testing
'TERMINATED for a task outside the scope of its master.) But the
individual compiler vendors maintain their own regression suites.

  > However, if I go to my customer, and say: "Our systems really
  > complex, and there's no way to develop a test suite that
  > guarantees bug-free operation, so you'll just have to live with
  > the current defect rate," he'll nod knowingly through the first
  > two statements, and cheerfully chop my head off after the
  > conclusion. That's the environment that I (and Mr McCabe, I
  > suspect) live in.
  
  How about you change the conclusion to "We can keep reducing the
defect rate, but we can never get it to zero."  You might get to keep
your head.

  > As a result, I have to define a means to reduce that error rate
  > over time. I have to measure that error rate, to show that it is
  > being reduced. And, (worst of all!) I have to share that error
  > rate with my customer. When the measured rate fails to meet my
  > goals, I get clobbered.  When it meets or exceeds my goals, do I
  > get flowers? No! But, at least I don't get clobbered.

   Exactly.

  > I understand that commercial software can and does work
  > differently. I also know that talking about a set of different,
  > competing companies as though they were a single entity ("the Ada
  > vendors") is also naive.

   Hmmm... SOME of the commercial software markets work much
differently. Others, especially in where systems are safety critical
look very much the same.

--

					Robert I. Eachus

with Standard_Disclaimer;
use  Standard_Disclaimer;
function Message (Text: in Clever_Ideas) return Better_Ideas is...