Ada Test Tools

[margono@grebyn.com (Johan Margono)]

I am the process of trying to locate possible testing tools to assist in
testing of Ada software. Some of my considerations are:

    1. The platforms that we are using...
          a. Rationals;
          b. MVS on IBM 3090s running TeleSoft Ada;
          c. AIX on IBM RS6000 running TeleSoft/OCS Ada.

    2. The greatest need is for Test Driver generators. A tool that will
       read in an Ada specification and generate an Ada program (with a
       user interface) that will allow a "tester" to call all operations
       and test exceptions, using his/her test cases.

    3. Test Case/Scenario generation tools would be nice.  Possible
       methods could include branch testing, McCabe's basis path method,
       and black-box testing (equivalence class partitioning and
       boundary analysis).

    4. Test tracking tools would be nice. Some tools assume an 'ad hoc'
       method for test case generation, and show branch coverage (so you
       can try a few more cases). Assuming a structured testing
       approach, this tracking does not seem very valuable.  However,
       tracking as a support for Regression Testing would be very
       helpful.

    5. Also related to (4), support for black-box testing of multiple
       instantiations of generics would be helpful (i.e., reuse of
       testing plans and test cases).

    6. Also related to (4) and (5), support for (at least) black-box
       testing of modules (generic/non-generic) on multiple platforms
       (rerunning all, or a subset, of the tests for each target).

    7. Many of the tools I have l looked at, depend on instrumentation
       of the code for test tracking, and possibly test generation.
       Since, instrumentation changes the 'code,' the load module, and
       thus behavior, this method is NOT preferred (it also becomes a
       maintenance headache). Non-intrusive methods are preferred!

    8. We are also restrained by 2167 (not A)!


I have collected information about several tools, but I am interested in
information on:
    1. additional tools not listed below,
    2. evaluations ("consumer reports") on any of the tools (listed or
       appended),
    3. (real) project experience on testing LARGE Ada systems, and the
       tools and techniques that helped/hindered.


----------------------- Current Testing Tool List ------------------------

1. Rational - public domain test driver generator which uses the
         Rational DIANA tree (available from the Rational Catalog). Need
         to check if the test drivers are portable to other platforms.
 
2. Nokia - (From Finland.)  TBGEN and TCMON; will run on the RAT and
         PCs.
 
3. Software Systems Design - TestGen does not have a driver generator.
         The test case generator is written in C so it won't run on the
         Rational.
 
4. McCabe Associates - no test driver, but other nice tools. I don't
         know if it will run on our platforms.

5. Intermetrics - Ada Test Support Tool (ATST) is public domain. It has a
          test driver, but only runs on VAX/VMS/ACS and PC/DOC/Alsys. It
          seems to only support black-box testing.
 
6. General Research Corporation - AdaQuest, was the Ada Test and
          Verification System (ATVS). Only VAX/VMS, with a UNIX platform
          later this year.
 
7. Glazier Electronics - The Automated Real-time Regression Testing
          system (ARRT). Only on VAX, only run time testing.
 
8. CISI Engineering - (From France) was a research project, that was not
          "productized"

--------------------- End Current Testing Tool List ---------------------


Snailmail: Gregory M. Bowen, Reuse Group
           Computer Sciences Corporation
           Advanced Automated System
           15245 Shady Grove Road
           Rockville, Maryland, 20850    USA

Phone:     301.921.3018
FAX:       301.840.8934
AdaNET ID: gbowen
Internet:  margono@grebyn.com   or  gbowen%ceo@asv2.wvnet.edu