Ada Test Tools

Ada Test Tools

[Lee McLoughlin]

[[RELAYED ON BEHALF OF anton@ist.co.uk - please reply directly to him.]]

Can anybody help me please ?

I am interested in commercially available test harnesses or coverage monitors
for Ada. Idealy, what I am after is something that is reasonably portable and
not too compiler specific.

If anyone knows of such things, then I would be most grateful to receive
details.

Thank you.

	 Anton Gibbs
	 Imperial Software Technology



E-mail:    anton@ist.co.uk or anton@ist.uucp
Telephone: +44 1 581 8155
Telex:     928476 istech g

Surface:   Imperial Software Technology Ltd.,
	   60 Albert Court,
	   Prince Consort Rd.,
	   London SW7 2BH
	   ENGLAND

==========================================================


----- End of forwarded messages

Re: Ada Test Tools

[Eric Beser sys admin]

The one test tool that we are using is the software analysis
workstation by Northwest Instruments. It is a logic analyzer for
the software engineer.

To call it a logic analyzer may be a bit unfair. It passively sits
on the processor and does monitor the logic states of the machine.
This device connects to the IBM PC/XT or AT by a parallel port that
is memory mapped into the system. Using the Gem operating system
by Digital Research (mouse, etc) it can monitor up to two processors.

It will take the symbol table output of a linker and use the symbols
for software analysis. We are using this tool to integrate the Telesoft
Ada Compiler with a comercial operating system on a distributed
68020 platform. As many of you know, the highest area of risk in this
venture is to solve the intricate timing problems involved with the Ada
RTE, hardware and application code. Because the analysis is symbolic
you get to see what portions of code execute, how much time it takes
to get through a procedure, the percentage of time spent in the RTE,
how interrupts are being serviced and dealt with in the RTE. It provides
2167 spec code coverage to validate the test procedures.

The only requirement for use is that the compiler puts out a symbol map
with enough symbols to do the job. Northwest Instruments supports their
own format, and the Unix Namelist (COFF). We have adapted the map
output of the Telesoft Linker, our own 1750A ada compiler, and various
Jovial compilers. It works well with C too (but we don't talk about
languages that ruin you as software engineers).

We have done a comparison study between the Software Analysis Workstation
and Hewlet Packard's 64000 workstation. Both are good. NWIS beats HP
on price and in many areas on performance as well. HP does statistical
analysis. With NWIS, you can time and size a program with one execution
of the program. 

Eric Beser
Westinghouse DEC
EBESER@ADA20   (arpa)
{seismo,mimsy}aplcen!cp1!sarin!eric  (usenet)

Ada Test Tools

[Johan Margono]

I am the process of trying to locate possible testing tools to assist in
testing of Ada software. Some of my considerations are:

    1. The platforms that we are using...
          a. Rationals;
          b. MVS on IBM 3090s running TeleSoft Ada;
          c. AIX on IBM RS6000 running TeleSoft/OCS Ada.

    2. The greatest need is for Test Driver generators. A tool that will
       read in an Ada specification and generate an Ada program (with a
       user interface) that will allow a "tester" to call all operations
       and test exceptions, using his/her test cases.

    3. Test Case/Scenario generation tools would be nice.  Possible
       methods could include branch testing, McCabe's basis path method,
       and black-box testing (equivalence class partitioning and
       boundary analysis).

    4. Test tracking tools would be nice. Some tools assume an 'ad hoc'
       method for test case generation, and show branch coverage (so you
       can try a few more cases). Assuming a structured testing
       approach, this tracking does not seem very valuable.  However,
       tracking as a support for Regression Testing would be very
       helpful.

    5. Also related to (4), support for black-box testing of multiple
       instantiations of generics would be helpful (i.e., reuse of
       testing plans and test cases).

    6. Also related to (4) and (5), support for (at least) black-box
       testing of modules (generic/non-generic) on multiple platforms
       (rerunning all, or a subset, of the tests for each target).

    7. Many of the tools I have l looked at, depend on instrumentation
       of the code for test tracking, and possibly test generation.
       Since, instrumentation changes the 'code,' the load module, and
       thus behavior, this method is NOT preferred (it also becomes a
       maintenance headache). Non-intrusive methods are preferred!

    8. We are also restrained by 2167 (not A)!


I have collected information about several tools, but I am interested in
information on:
    1. additional tools not listed below,
    2. evaluations ("consumer reports") on any of the tools (listed or
       appended),
    3. (real) project experience on testing LARGE Ada systems, and the
       tools and techniques that helped/hindered.


----------------------- Current Testing Tool List ------------------------

1. Rational - public domain test driver generator which uses the
         Rational DIANA tree (available from the Rational Catalog). Need
         to check if the test drivers are portable to other platforms.
 
2. Nokia - (From Finland.)  TBGEN and TCMON; will run on the RAT and
         PCs.
 
3. Software Systems Design - TestGen does not have a driver generator.
         The test case generator is written in C so it won't run on the
         Rational.
 
4. McCabe Associates - no test driver, but other nice tools. I don't
         know if it will run on our platforms.

5. Intermetrics - Ada Test Support Tool (ATST) is public domain. It has a
          test driver, but only runs on VAX/VMS/ACS and PC/DOC/Alsys. It
          seems to only support black-box testing.
 
6. General Research Corporation - AdaQuest, was the Ada Test and
          Verification System (ATVS). Only VAX/VMS, with a UNIX platform
          later this year.
 
7. Glazier Electronics - The Automated Real-time Regression Testing
          system (ARRT). Only on VAX, only run time testing.
 
8. CISI Engineering - (From France) was a research project, that was not
          "productized"

--------------------- End Current Testing Tool List ---------------------


Snailmail: Gregory M. Bowen, Reuse Group
           Computer Sciences Corporation
           Advanced Automated System
           15245 Shady Grove Road
           Rockville, Maryland, 20850    USA

Phone:     301.921.3018
FAX:       301.840.8934
AdaNET ID: gbowen
Internet:  margono@grebyn.com   or  gbowen%ceo@asv2.wvnet.edu

RE: Ada Test Tools

[Bill Brykczynski]

The following is in response to a recent request for information
regarding tools to assist in the testing of Ada software.  We are just
beginning to collect such information.  In the future, we may be doing
an evaluation of several of these tools.  This information is not
currently not complete, but may still be of use.

We are not recommending any of these tools, we are simply collecting
information on them.  If you have additional information on these or
other tools, experience with such tools, etc., please let us know.
As you can see, some of the information is skimpy.

Bill Brykczynski
Institute for Defense Analyses
1801 N. Beauregard St.
Alexandria, VA 22311
703-845-6641
bryk@ida.org 
--------------
                                     - 1 - 
 
 
 
          Alsys, Inc. 
 
                    Tool:          AdaTune, AdaProbe 
                    Contact:       Lori Heyman (617) 270-0030 
                                   One Burlington Business Center 
                                   67 South Bedford Street, Burlington,  MA 
                                   01803-5152 
 
          Application Development System, Inc. 
 
                    Tool:          XPEDITER (Test Driver) 
                    Contact:       E.F. Harris 
                                   1530 Meridian Avenue, San Jose, CA 95125 
 
          Cadre Technologies 
 
                    Tool:          Software  Analysis  Workstation   (SAW), 
                                   real-time performance measurement. 
                    Tool:          CodeMap,  tracking  tool  to  see  which 
                                   software parts are exercised. 
                    Tool:          XRAY, simulator/debugger to execute code 
                                   on workstation (prior to target macine). 
                                   Rapid prototyping. 
                    Tool:          MDD,  design  phase  analysis  to  check 
                                   performance. 
                    Tool:          SoftAnalyst, Sym Trace provides symbolic 
                                   trace of program and data flow. 
                    Contact:       Myra Linch (703) 875-8670 
 
          Convex Computer Corp. 
 
                    Tool  Data:     Interactive  performance  profilers,  & 
                                   Verdix development environment. 
                    Contact:       Larry Grossman (214) 497-4383 
                                   3000 Waterview Pkwy,  P.O.  Box  883851, 
                                   Richardson, TX 78053-5851 
 
          DDC-1 
 
 
 
                                     - 2 - 
 
 
 
                    Tool:          Symbolic  Ada  debugger,  non-intrusive, 
                                   can input file of test data 
                    Contact:       Forest Holeman (602) 944-1883 
 
          Dynamics Research Corp. 
 
                    Tool:          AdaMAT 
                    Contract:      John Ragosta (508) 475-9090 
                                   60 Frontage Road, Andover, MA 01810 
 
          EVB Software Engineering Inc. 
 
                    Tool:          Complexity measurement 
                    Contact:       Jennifer Lot (301) 695-6960 
                                   5320 Spectrum Drive, Frederick, MD 21701 
 
          Generics Software Ltd. 
 
                    Tool:          AnimAda 
                    Contact:       +353-1-954012 
                                   7  Leopardstown  Office  Park,  Foxrock, 
                                   Dublin 18, Ireland 
 
          INRIA 
 
                    Tool:          Mentor-Ada, measurement system 
                    Contact:       Schroeder A. 
                                   Domaine   de   Voluceau,    B.P.    105, 
                                   Rocquencourt, 78153 Le Chesnay, France 
 
          ITT Avionics 
 
                    Tool Data:     Funded by STARS 
 
                                   Tool: UATL: Universal Ada Test  Language 
                                        (Test Driver) 
                    Contact:       Ziegler J.  (201) 284-5030 
                                   390  Washington   Avenue,   Nutley,   NJ 
                                   070110-5603 
 
 
 
                                     - 3 - 
 
 
 
          Jackson Systems Corporation 
 
                    Tool Data:     data against data and program structures 
                                   to   verify    structure    correctness. 
                                   Structures  may  be  derived  form  test 
                                   data. Test  data  may  be  generated  to 
                                   create templates for program testing. 
                    "Contact:"  Michael Jackson Systems Limited, London 
 
          Programming Environments 
 
                    Tool:          T test case generator 
                    Description:   Verifies    requirements    testability, 
                                   produces  test   cases,   and   analyzes 
                                   testing comprehensiveness.   Inputs  and 
                                   outputs are documented, flat ASCII files 
                                   to allow working with descriptions  from 
                                   other   tools.    Designs   test   cases 
                                   automatically  or  provides  help  in  a 
                                   computer-aided mode, for 100% functional 
                                   coverage and  100%  most-probable-error- 
                                   coverage.  Runs under MS-DOS, UNIX,  and 
                                   VAX/VMS.  Single  prince  copy  for  MS- 
                                   DOS/UNIX systems is $7000. 
                    Contact:       Robert Poston work (201) 918-0110,  home 
                                   (201) 922-0968 
                                   PE Inc. 
                                   4043 State Highway 33, Tinton Falls,  NJ 
                                   07753 
 
          SQL Systems International 
 
                    Tool:          PCMS*ADA, dependency analysis 
                    Contact:       England, 44-279-641021 
 
          Set Labs 
 
                    Tool:          UX-metric for  estimates  of  errors  in 
 
 
 
                                     - 4 - 
 
 
 
                                   design. Support for Ada. 
                    Contact:       Teresa Harrison (503) 289-4758 
 
          Software Research 
 
                    Tool Data:     Full Ada support,  several  environments 
                                   including Sun. 
 
                                   Tool: Capbak/UNIX, capture and  playback 
                                        of test sessions. 
                                   Tool: EXDIFF    (Extended     Difference 
                                        Analyzer) 
                                   Tool: SMARTS (Software  Maintenance  and 
                                        Regression Testing Suite) 
                                   Tool: STCAT/C  (System   Test   Coverage 
                                        Analysis    Tool),    function-call 
                                        coverage. 
                                   Tool: TCAT/C  (Test  Coverage   Analysis 
                                        Tool) 
                                   Tool: TCAT/Path,  profiles   paths   and 
                                        produces graph. 
                                   Tool: TDGen, test data generation 
                    Contact:       Frank Vimaggi? (415) 957-1441 
 
          Teledyne Brown Engineering 
 
                    Tool Data: 
 
                                   Tool: ACAT: Ada Complexity Analysis Tool 
                                   Tool: TAGS/CASE2,  diagnostic   analyzer 
                                        provides static analysis to  ensure 
                                        IORL   syntax   is   complete   and 
                                        correct, and analysis library 
                    Contact:       Mike Cario 352-8533 
 
          TeleSoft 
 
                    Tool:          Telegen2      (Telesoft      development 
                                   environment), browser/profiler 
 
 
 
                                     - 5 - 
 
 
 
                    Contact:       Jeff Kelley (619) 457-2700 
                                   5959 Cornerstone Court West, San  Diego, 
                                   CA 92121 
 
          Verilog SA 
 
                    Tool Data: 
 
                                   Tool: LOGISCOPE, control flow testing at 
                                        unit   and    integration    level, 
                                        currently DD-path and PP-path, some 
                                        to include LCSAJ, graphical display 
                                        of   coverage    with    cumulative 
                                        coverage reporting, full Ada. 
                                   Tool: ASA,  support  for   SADT   IDEF0, 
                                        simluation and test case generation 
                                        for validation of requirements  and 
                                        validation   of   system    against 
                                        requirements. 
                                   Tool: GEODE,    supports    CCITT    SDL 
                                        language, in 6 - 8 months  expected 
                                        to generate Ada code. 
                    Contact:       Nandan  Shah  (technical),  Mark  Luciw, 
                                   Susan Weil (sales) (703) 354-0371 

Re: Ada Test Tools

[Scott Layson]

I am implementing an Ada front end (more or less) and I have
come across a rule in the (mostly very precise) manual that
seems unclear.  Perhaps someone can help me with it.

It has to do with paragraphs 18 and 19 of section 4.1.3.
(Please refer to the manual at this point.)  Paragraph 17 talks
about four kinds of named constructs whose names can be used as
the prefix of an expanded name.  Paragraphs 18 and 19 give
additional rules about only two of those kinds of constructs,
viz., subprograms and accept statements.  What about block
statements and loop statements?

Thanks in advance,

Scott Burson
Gyro@Reasoning.COM

Ada test tools

[d]

Looking into some test tools like AdaCAST (from VectorCAST) and also
LDRA.  Unit testing, MC/DC, integration level tests, coverage
analysis, auto test driver a plus.

What is the current state of test tools using primarily Ada?

Suggestions?

Cheers

Re: Ada test tools

[R. Tim Coslet]

I've used AdaCAST quite a bit.

Overall it seems to work very well for CUT level testing.

However the initial versions we got had severe problems testing packages
that contained tasks. I spent ALOT of time sending them sample test
environments that would not work correctly to get various bugs fixed.
Current versions seem to be working reasonably well for us.


Another one you could look at is AdaTest (from a British company, I don't
recall their name). What I see of their ads they look more "stable" and
"mature" than AdaCAST, but I've never had a chance to try it out.


-- 
        R. Tim Coslet
        r_tim_coslet@pacbell.net

Technology, n. Domesticated natural phenomena.


> From: coolpages@excite.com (d)
> Organization: http://groups.google.com/
> Newsgroups: comp.lang.ada
> Date: 10 Sep 2002 12:59:29 -0700
> Subject: Ada test tools
> 
> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.
> 
> What is the current state of test tools using primarily Ada?
> 
> Suggestions?
> 
> Cheers

Re: Ada test tools

[John Kern]

How about TestRT from Rational Corporation.
http://www.rational.com/products/testrt/index.jsp

d wrote:
> 
> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.
> 
> What is the current state of test tools using primarily Ada?
> 
> Suggestions?
> 
> Cheers

Re: Ada test tools

[Peter Amey]

d wrote:
> 
> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.
> 
> What is the current state of test tools using primarily Ada?
> 
> Suggestions?
> 

AdaTest from IPL (http://www.iplbath.com/products/tools/pt600.shtml)

Peter

Re: Ada test tools

[Simon Wright]

coolpages@excite.com (d) writes:

> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.

We are using AUnit from ACT, it doesn't do all the fancy stuff (MC/DC
etc) but it is working very well.

Also gcov for coverage; not always completely straightforward,
especially with generics, but still useful.

Re: Ada test tools

[Colin Paul Gloster]

In article <B9A41295.52F2%R_Tim_Coslet@pacbell.net>, R. Tim Coslet wrote:
"[..]  

Another one you could look at is AdaTest (from a British company, I don't
recall their name). [..]"

Peter Amey already posted about IPL before I saw your message, but I can
add to the thread a mention of an exclusively coverage testing aid from
elsewhere in the British Isles: HTTP://Glasnost.ITCarlow.Ie/~meudecc/atgen/
(Christophe Meudec, +353 503 76266).

Re: Ada test tools

[Patricia Langle]

As Vector Software partner, Aonix France has now a good experience of
using VectorCAST/Ada on big projects.
Our professional services and our customers have successfully tested Ada
applications. The last one was for SIL4 certification (highest critical
level in ground transportation systems).
Our teams appreciate the tools (user friendly) and the efficiency of
Vector Software customer support.
Just have a look on their Web site www.vectorcast.com, you could
download all the documentation you need.

Best regards

Patricia


d a �crit :

> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.
>
> What is the current state of test tools using primarily Ada?
>
> Suggestions?
>
> Cheers

Re: Ada test tools

[KSW]

d- I would be happy to review our unit testing and code coverage product
suite with you.  Feel free to contact me.
Kyle Wilbur
Vector Software
ksw@vectors.com
(401)295-5855 x117

"d" <coolpages@excite.com> wrote in message
news:f2652562.0209101159.4b4ec60b@posting.google.com...
> Looking into some test tools like AdaCAST (from VectorCAST) and also
> LDRA.  Unit testing, MC/DC, integration level tests, coverage
> analysis, auto test driver a plus.
>
> What is the current state of test tools using primarily Ada?
>
> Suggestions?
>
> Cheers