Re: Automatic Test Code Generator

[Ken Garlington <garlingtonke@lmtas.lmco.com>]

Larry Kilgallen wrote:
> 
> In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes:
> > I'm trying to find a test case generator for Ada.
> 
> > [2] The test cases should be able to exercise
> >     each instruction (not each path).
> 
> Hmmm.
> 
>         a) I don't see how any test case generator could exercise
>            a single instruction without exercising other instructions
>            on that path.

I think he's saying that he wants statement coverage, which is a weaker 
criteria than path coverage. For example, consider the following code:

for I in 1 .. Input loop
  if Input = 1 then
    Do_Something;
  end if;
  Do_Something_Else;
end loop;

As I usually explain the distinctions:

(a) One test case (with Input=1) is sufficient for statement coverage.
    Each statement above is visited in the execution of the loop.

(b) Two tests cases (with Input=1 and Input > 1, respectively) are
    needed for branch coverage. This will visit each statement
    (statement coverage), and will also exercise every outcome of every
    branch. Thus, branch coverage is a superset of statement coverage.

(c) Multiple test cases (with Input set to each valid value) are needed
    for path coverage. This exercises every possible set of statements,
    and so is a superset of either branch coverage or statement
    coverage. The downside, of course, is that you may end up with a
    very large number of tests.

Note that statement coverage is not entirely insensitive to branches,
however. For example, given the code

  if Input = 1 then
    Do_Something;
  else
    Do_Something_Else;
  end if;

then two test cases are needed just to achieve statement coverage.

> I don't think what you are looking for would be possible as an
> Ada-centric tool, since if you are concerned about code paths
> and individual instructions you are down below the Ada syntax
> to the level of the particular code generator used by your
> compiler.

If, by "statement", the original request was referring to Ada 
statements, there are probably test case generators available.
Check the AdaIC web page at

  http://sw-eng.falls-church.va.us/AdaIC/tools/Welcome.html

I know there are several tools available that can _identify_ the
tests that need to be run, but I don't know how many will actually
generate the test harnesses directly from that analysis. Usually,
you have to have some form of data flow analysis, along with the
structural analysis, to determine which inputs to select for each
test to get the desired level of coverage.

To do this at the object level is more difficult, although not
impossible. We have custom tools to do this for the MIL-STD-1750A
(the analysis, not the test case generation, although the generation
is also possible).

-- 
LMTAS - "Our Brand Means Quality"
For more info, see http://www.lmtas.com or http://www.lmco.com