* Automatic Test Code Generator @ 1996-11-26 0:00 Art Schwarz 1996-11-26 0:00 ` Larry Kilgallen ` (2 more replies) 0 siblings, 3 replies; 6+ messages in thread From: Art Schwarz @ 1996-11-26 0:00 UTC (permalink / raw) I'm trying to find a test case generator for Ada. The minimum requirements are: [1] Be able to generate test cases for a single procedure. [2] The test cases should be able to exercise each instruction (not each path). I suspect that I could modify the static code analyser (in "The Home of the Brave Ada Programmers" or Arcadia) so that it could generate test cases but, if such a tool is available I'd rather use it (software reuse being what it is these days). Anyone have a clue as to availability? thanks art schwarz General Dynamic Land Systems (810)825-5028 schwarza@gdls.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic Test Code Generator 1996-11-26 0:00 Automatic Test Code Generator Art Schwarz @ 1996-11-26 0:00 ` Larry Kilgallen 1996-11-27 0:00 ` Ken Garlington 1996-11-27 0:00 ` Stephen Leake 1996-11-27 0:00 ` Ken Garlington 2 siblings, 1 reply; 6+ messages in thread From: Larry Kilgallen @ 1996-11-26 0:00 UTC (permalink / raw) In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes: > I'm trying to find a test case generator for Ada. > [2] The test cases should be able to exercise > each instruction (not each path). Hmmm. a) I don't see how any test case generator could exercise a single instruction without exercising other instructions on that path. b) I don't see how any test case generator which exercised all paths could possibly miss any instructions. I don't think what you are looking for would be possible as an Ada-centric tool, since if you are concerned about code paths and individual instructions you are down below the Ada syntax to the level of the particular code generator used by your compiler. For VMS (and presumably many other operating systems) you can easily (given appropriate debug symbol table entries) _check_ whether your test suite covers all code paths, but automatically generating such a test suite which covers all code paths seems to me a formidable task. Larry Kilgallen ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic Test Code Generator 1996-11-26 0:00 ` Larry Kilgallen @ 1996-11-27 0:00 ` Ken Garlington 1996-11-27 0:00 ` Larry Kilgallen 0 siblings, 1 reply; 6+ messages in thread From: Ken Garlington @ 1996-11-27 0:00 UTC (permalink / raw) Larry Kilgallen wrote: > > In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes: > > I'm trying to find a test case generator for Ada. > > > [2] The test cases should be able to exercise > > each instruction (not each path). > > Hmmm. > > a) I don't see how any test case generator could exercise > a single instruction without exercising other instructions > on that path. I think he's saying that he wants statement coverage, which is a weaker criteria than path coverage. For example, consider the following code: for I in 1 .. Input loop if Input = 1 then Do_Something; end if; Do_Something_Else; end loop; As I usually explain the distinctions: (a) One test case (with Input=1) is sufficient for statement coverage. Each statement above is visited in the execution of the loop. (b) Two tests cases (with Input=1 and Input > 1, respectively) are needed for branch coverage. This will visit each statement (statement coverage), and will also exercise every outcome of every branch. Thus, branch coverage is a superset of statement coverage. (c) Multiple test cases (with Input set to each valid value) are needed for path coverage. This exercises every possible set of statements, and so is a superset of either branch coverage or statement coverage. The downside, of course, is that you may end up with a very large number of tests. Note that statement coverage is not entirely insensitive to branches, however. For example, given the code if Input = 1 then Do_Something; else Do_Something_Else; end if; then two test cases are needed just to achieve statement coverage. > I don't think what you are looking for would be possible as an > Ada-centric tool, since if you are concerned about code paths > and individual instructions you are down below the Ada syntax > to the level of the particular code generator used by your > compiler. If, by "statement", the original request was referring to Ada statements, there are probably test case generators available. Check the AdaIC web page at http://sw-eng.falls-church.va.us/AdaIC/tools/Welcome.html I know there are several tools available that can _identify_ the tests that need to be run, but I don't know how many will actually generate the test harnesses directly from that analysis. Usually, you have to have some form of data flow analysis, along with the structural analysis, to determine which inputs to select for each test to get the desired level of coverage. To do this at the object level is more difficult, although not impossible. We have custom tools to do this for the MIL-STD-1750A (the analysis, not the test case generation, although the generation is also possible). -- LMTAS - "Our Brand Means Quality" For more info, see http://www.lmtas.com or http://www.lmco.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic Test Code Generator 1996-11-27 0:00 ` Ken Garlington @ 1996-11-27 0:00 ` Larry Kilgallen 0 siblings, 0 replies; 6+ messages in thread From: Larry Kilgallen @ 1996-11-27 0:00 UTC (permalink / raw) In article <329C05F3.182C@lmtas.lmco.com>, Ken Garlington <garlingtonke@lmtas.lmco.com> writes: > Larry Kilgallen wrote: >> >> In article <57ffin$il@mill.gdls.com>, schwarza@gdls.com (Art Schwarz) writes: >> > I'm trying to find a test case generator for Ada. >> >> > [2] The test cases should be able to exercise >> > each instruction (not each path). > If, by "statement", the original request was referring to Ada > statements, there are probably test case generators available. But the term I quoted was "instruction", which to me indicates he was talking about the machine language level. And in self-defense against any language lawyers in the crowd, I just looked up "instruction" in the Ada95 RM Index and did not find it had some other Ada-specific meaning :-). Of course if he means "statement" then addressing the problem at the Ada level is tractable (though perhaps only by human effort). Larry Kilgallen ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic Test Code Generator 1996-11-26 0:00 Automatic Test Code Generator Art Schwarz 1996-11-26 0:00 ` Larry Kilgallen @ 1996-11-27 0:00 ` Stephen Leake 1996-11-27 0:00 ` Ken Garlington 2 siblings, 0 replies; 6+ messages in thread From: Stephen Leake @ 1996-11-27 0:00 UTC (permalink / raw) We looked into this recently. We found two commercial vendors (there are probably others): ** McCabe Associates 800-638-6316 http://www.mccabe.com ** Vector Software Inc. 401-295-5855 1130 Ten Rod Road Suite E-307 North Kingstown, Rhode Island 02852 fax: 401-295-5856 www.vectors.com McCabe makes a static path analysis tool, and a path coverage reporting tool (it instruments your code and gathers statistics). Vector makes a test suite generator. Both say generating the actual test case data is up to you; they wish they knew how to do it automatically - they'd get rich! The McCabe tool will help a lot in generating the test cases, and in verifying that you have actually covered each branch at the Ada source level. As another poster pointed out, it is not clear that this is the same as covering each instruction, particularly since adding the reporting code can affect what the compiler optimizer does. -- - Stephe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic Test Code Generator 1996-11-26 0:00 Automatic Test Code Generator Art Schwarz 1996-11-26 0:00 ` Larry Kilgallen 1996-11-27 0:00 ` Stephen Leake @ 1996-11-27 0:00 ` Ken Garlington 2 siblings, 0 replies; 6+ messages in thread From: Ken Garlington @ 1996-11-27 0:00 UTC (permalink / raw) Art Schwarz wrote: > > I'm trying to find a test case generator for Ada. > The minimum requirements are: > [1] Be able to generate test cases for a single > procedure. > [2] The test cases should be able to exercise > each instruction (not each path). > > I suspect that I could modify the static code analyser > (in "The Home of the Brave Ada Programmers" or Arcadia) > so that it could generate test cases but, if such a > tool is available I'd rather use it (software reuse > being what it is these days). Anyone have a clue as > to availability? Check the AdaIC web page at http://sw-eng.falls-church.va.us/AdaIC/tools/Welcome.html I know there are several tools available that can _identify_ the tests that need to be run, but I don't know how many will actually generate the test harnesses directly from that analysis. We usually generate our tests from the functional requirements, then use the coverage analysis as a check on the quality of the tests, rather than generating tests directly from the analysis. > > thanks > > art schwarz > General Dynamic Land Systems > (810)825-5028 > schwarza@gdls.com -- LMTAS - "Our Brand Means Quality" For more info, see http://www.lmtas.com or http://www.lmco.com ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~1996-11-27 0:00 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 1996-11-26 0:00 Automatic Test Code Generator Art Schwarz 1996-11-26 0:00 ` Larry Kilgallen 1996-11-27 0:00 ` Ken Garlington 1996-11-27 0:00 ` Larry Kilgallen 1996-11-27 0:00 ` Stephen Leake 1996-11-27 0:00 ` Ken Garlington
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox