Automatic Test Code Generator

Automatic Test Code Generator

[Art Schwarz]

I'm trying to find a test case generator for Ada.
The minimum requirements are:
[1] Be able to generate test cases for a single 
    procedure.
[2] The test cases should be able to exercise
    each instruction (not each path).

I suspect that I could modify the static code analyser
(in "The Home of the Brave Ada Programmers" or Arcadia)
so that it could generate test cases but, if such a
tool is available I'd rather use it (software reuse 
being what it is these days). Anyone have a clue as
to availability?

thanks

art schwarz
General Dynamic Land Systems
(810)825-5028
schwarza@gdls.com

Re: Automatic Test Code Generator

[Larry Kilgallen]

In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes:
> I'm trying to find a test case generator for Ada.

> [2] The test cases should be able to exercise
>     each instruction (not each path).

Hmmm.

	a) I don't see how any test case generator could exercise
	   a single instruction without exercising other instructions
	   on that path.

	b) I don't see how any test case generator which exercised
	   all paths could possibly miss any instructions.

I don't think what you are looking for would be possible as an
Ada-centric tool, since if you are concerned about code paths
and individual instructions you are down below the Ada syntax
to the level of the particular code generator used by your
compiler.  For VMS (and presumably many other operating systems)
you can easily (given appropriate debug symbol table entries)
_check_ whether your test suite covers all code paths, but
automatically generating such a test suite which covers all
code paths seems to me a formidable task.

Larry Kilgallen

Re: Automatic Test Code Generator

[Ken Garlington]

Larry Kilgallen wrote:
> 
> In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes:
> > I'm trying to find a test case generator for Ada.
> 
> > [2] The test cases should be able to exercise
> >     each instruction (not each path).
> 
> Hmmm.
> 
>         a) I don't see how any test case generator could exercise
>            a single instruction without exercising other instructions
>            on that path.

I think he's saying that he wants statement coverage, which is a weaker 
criteria than path coverage. For example, consider the following code:

for I in 1 .. Input loop
  if Input = 1 then
    Do_Something;
  end if;
  Do_Something_Else;
end loop;

As I usually explain the distinctions:

(a) One test case (with Input=1) is sufficient for statement coverage.
    Each statement above is visited in the execution of the loop.

(b) Two tests cases (with Input=1 and Input > 1, respectively) are
    needed for branch coverage. This will visit each statement
    (statement coverage), and will also exercise every outcome of every
    branch. Thus, branch coverage is a superset of statement coverage.

(c) Multiple test cases (with Input set to each valid value) are needed
    for path coverage. This exercises every possible set of statements,
    and so is a superset of either branch coverage or statement
    coverage. The downside, of course, is that you may end up with a
    very large number of tests.

Note that statement coverage is not entirely insensitive to branches,
however. For example, given the code

  if Input = 1 then
    Do_Something;
  else
    Do_Something_Else;
  end if;

then two test cases are needed just to achieve statement coverage.

> I don't think what you are looking for would be possible as an
> Ada-centric tool, since if you are concerned about code paths
> and individual instructions you are down below the Ada syntax
> to the level of the particular code generator used by your
> compiler.

If, by "statement", the original request was referring to Ada 
statements, there are probably test case generators available.
Check the AdaIC web page at

  http://sw-eng.falls-church.va.us/AdaIC/tools/Welcome.html

I know there are several tools available that can _identify_ the
tests that need to be run, but I don't know how many will actually
generate the test harnesses directly from that analysis. Usually,
you have to have some form of data flow analysis, along with the
structural analysis, to determine which inputs to select for each
test to get the desired level of coverage.

To do this at the object level is more difficult, although not
impossible. We have custom tools to do this for the MIL-STD-1750A
(the analysis, not the test case generation, although the generation
is also possible).

-- 
LMTAS - "Our Brand Means Quality"
For more info, see http://www.lmtas.com or http://www.lmco.com

Re: Automatic Test Code Generator

[Larry Kilgallen]

In article <329C05F3.182C@lmtas.lmco.com>, Ken Garlington <garlingtonke@lmtas.lmco.com> writes:
> Larry Kilgallen wrote:
>> 
>> In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes:
>> > I'm trying to find a test case generator for Ada.
>> 
>> > [2] The test cases should be able to exercise
>> >     each instruction (not each path).

> If, by "statement", the original request was referring to Ada 
> statements, there are probably test case generators available.

But the term I quoted was "instruction", which to me indicates he
was talking about the machine language level.

And in self-defense against any language lawyers in the crowd,
I just looked up "instruction" in the Ada95 RM Index and did
not find it had some other Ada-specific meaning :-).

Of course if he means "statement" then addressing the problem at
the Ada level is tractable (though perhaps only by human effort).

Larry Kilgallen

Re: Automatic Test Code Generator

[Ken Garlington]

Art Schwarz wrote:
> 
> I'm trying to find a test case generator for Ada.
> The minimum requirements are:
> [1] Be able to generate test cases for a single
>     procedure.
> [2] The test cases should be able to exercise
>     each instruction (not each path).
> 
> I suspect that I could modify the static code analyser
> (in "The Home of the Brave Ada Programmers" or Arcadia)
> so that it could generate test cases but, if such a
> tool is available I'd rather use it (software reuse
> being what it is these days). Anyone have a clue as
> to availability?

Check the AdaIC web page at

  http://sw-eng.falls-church.va.us/AdaIC/tools/Welcome.html

I know there are several tools available that can _identify_ the
tests that need to be run, but I don't know how many will actually
generate the test harnesses directly from that analysis. We usually
generate our tests from the functional requirements, then use the
coverage analysis as a check on the quality of the tests, rather than
generating tests directly from the analysis.

> 
> thanks
> 
> art schwarz
> General Dynamic Land Systems
> (810)825-5028
> schwarza@gdls.com

-- 
LMTAS - "Our Brand Means Quality"
For more info, see http://www.lmtas.com or http://www.lmco.com

Re: Automatic Test Code Generator

[Stephen Leake]

We looked into this recently. We found two commercial vendors (there are
probably others):

** McCabe Associates  800-638-6316
http://www.mccabe.com

** Vector Software Inc. 401-295-5855
     1130 Ten Rod Road
     Suite E-307
     North Kingstown, Rhode Island 02852
     fax:   401-295-5856
    www.vectors.com

McCabe makes a static path analysis tool, and a path coverage reporting
tool (it instruments your code and gathers statistics). Vector makes a
test suite generator.

Both say generating the actual test case data is up to you; they wish
they knew how to do it automatically - they'd get rich!

The McCabe tool will help a lot in generating the test cases, and in
verifying that you have actually covered each branch at the Ada source
level. As another poster pointed out, it is not clear that this is the
same as covering each instruction, particularly since adding the
reporting code can affect what the compiler optimizer does.

-- 
- Stephe