RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[David Akister]

I would like to know if anyone  can share experience of using the Rational
Testmate tool with Rational Apex on UNIX. Of particular interest is the
management of code units, time taken developing and modifying test-sets and
flexibility of the system.

Oh, and before I forget how does the system compare with other test suites
for ADA testing such as ADATEST and LDRA (for coverage metrics) ?

Any help is much appreciated
Regards

Dave Akister
Senior Software Engineer
Wellgate Technology Ltd

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Ray Blaak]

"David Akister" <dakister@compd.com> writes:

> I would like to know if anyone  can share experience of using the Rational
> Testmate tool with Rational Apex on UNIX. Of particular interest is the
> management of code units, time taken developing and modifying test-sets and
> flexibility of the system.

I have used it on a large air traffic control project, and found it quite
useful for unit testing within a subsystem. One can test against expected
outputs, or write general test drivers that set pass/fail explicitly.  If one
controls the test outputs under CMVC, then one has a nice historical record of
how tests pass/fail.

Time taken developing tests is rather dependent on the unit being tested,
rather than testmate itself (basically, testmate doesn't really get in the
way). Testing code should be considered as official software units that get
carefully developed with future maintenance and enhancements in mind, reviewed
as part of the unit, and need to be budgeted accordingly. (The biggest mistake
I saw was to see testing code written as if it was throw-away code -- it never
was, and all too often cut/paste/spaghetti code need to be enhanced to test new
unit changes).

The best feature of testmate that I liked was the code coverage analysis, which
highlights those code regions not executed in a test.

Basically, testmate is a good tool, but one also needs a good development
process to enforce 1) that things are tested at all, 2) how they are tested
(i.e. what information needs to be in the test output, and what is deemed
sufficient coverage, etc.), and 3) what kinds of test information needs to be
saved.

> Oh, and before I forget how does the system compare with other test suites
> for ADA testing such as ADATEST and LDRA (for coverage metrics) ?

I have not used them.

-- 
Cheers,                                        The Rhythm is around me,
                                               The Rhythm has control.
Ray Blaak                                      The Rhythm is inside me,
blaak@infomatch.com                            The Rhythm has my soul.

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Julius Lancer]

In article <m3aeuwsle6.fsf@ns32.infomatch.bc.ca>, Ray Blaak <blaak@infomatch.com> wrote:

>Basically, testmate is a good tool, but one also needs a good development
>process to enforce 1) that things are tested at all, 2) how they are tested
>(i.e. what information needs to be in the test output, and what is deemed
>sufficient coverage, etc.), and 3) what kinds of test information needs to be
>saved.

Sounds like some mouse needs to tie the bell around the cat's neck.

John - N8086N 
Big brother is watching. Disable cookies in your web browser.
-------------------------------------------
Wise man says "Never use a bank with the initials F. U."
-------------------------------------------
Are you interested in a professional society or
guild for programmers? Want to fight section 1706?


See www.programmersguild.org
Newsgroup: us.issues.occupations.computer-programmers


EMail Address:
_m-i-a-n-o_@_c_o_l_o_s_s_e_u_m_b_u_i_l_d_e_r_s._c_o_m_

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Brian Collins]

Dave,

I have used the Testmate MCDC tool on many verification projects involving
embedded avionics software.  If your code uses the Rational Ada compiler and
the workstation you are using it on is both the host and target environment,
then everything is very cool and this is a very slick tool to use.  When you
deviate from those two rules the tool continues to operate, but it will take
a lot more  effort to keep it going.  You can still keep the test cases in
Testmate and keep track of your verification effort, but it becomes much
less automated.  In our environment we have to deviate from both rules by
first using a non-Apex cross-compiler via RCI and by second having our
target environment in our actual hardware box and not on the workstation.
Breaking the first rule causes us some grief sometimes with code that simply
will not instrument, but this is pretty rare and usually has to do with the
differences between the cross-compiler's understanding of Ada and Apex's
understanding of Ada.  Breaking the second rule is what hurts us the most by
not having the test cases automated which takes the most time to create each
test case and test by hand all of our decisions and conditions in our code
then giving Testmate the results.  If you have an automated way of runing
through your test cases with scripts and collecting coverage, then you will
really like this tool.  Where can I get information on the other tools you
mentioned?  Thank you.

Brian Collins
brian.collins@www.mebbs.com
bjcollin@rockwell.collins.com
http://www.math.swt.edu/~rambo

David Akister wrote in message <374667df@news.compd.com>...
>I would like to know if anyone  can share experience of using the Rational
>Testmate tool with Rational Apex on UNIX. Of particular interest is the
>management of code units, time taken developing and modifying test-sets and
>flexibility of the system.
>
>Oh, and before I forget how does the system compare with other test suites
>for ADA testing such as ADATEST and LDRA (for coverage metrics) ?
>
>Any help is much appreciated
>Regards
>
>Dave Akister
>Senior Software Engineer
>Wellgate Technology Ltd
>
>
>
>

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[David Akister]

Brian,
Thanks for your response, I think you hit the nail on the head certainly
with my impressions of the TESTMATE suite. It seems well integrated with the
Apex environment, but the need to cross-compile on a non-Rational target
compiler means that a lot of the advantage is lost. With the need for
validated compilers for safety-critical use this does make life difficult.
Additionally one of the main selling points appears to be the ability to
provide code coverage listings based on 'Persistant Intermediate
representations' ie object code. This is is good stuff if you're using the
prescribed compiler....

Generally though the demonstration that I was party to was very impressive,
and generation of simple test cases and stubs could be usefully
auto-generated. Test-case replicators allow test-lists to be easily produced
from the test case template. All good stuff in my opinion... a lot of time
is usually spent manually reproducing test cases with varying parameters and
managing them.This makes things pretty easy.

Overall then, assuming that the end customer is happy with the choice of
cross-compiler, it has my recommendation (for what its worth !)

LDRA stands for Liverpool Data Research Associates and they produce a very
good suite of tools for static analysis and dynamic code coverage. This
generates some very useful complexity and knotting reports but in my
experience it's used primarily for code coverage, with alternate forms of
static analysis tools are used for assessment and possible proving. examples
of these are Spark (excellent) and MALPAS (good output but heavy going).

If you're interested, I'll try and dig out some contact numbers.

Regards
David Akister

Brian Collins wrote in message <7i99ls$lv$1@ins8.netins.net>...
>Dave,
>
>I have used the Testmate MCDC tool on many verification projects involving
>embedded avionics software.  If your code uses the Rational Ada compiler
and
>the workstation you are using it on is both the host and target
environment,
>then everything is very cool and this is a very slick tool to use.  When
you
>deviate from those two rules the tool continues to operate, but it will
take
>a lot more  effort to keep it going.  You can still keep the test cases in
>Testmate and keep track of your verification effort, but it becomes much
>less automated.  In our environment we have to deviate from both rules by
>first using a non-Apex cross-compiler via RCI and by second having our
>target environment in our actual hardware box and not on the workstation.
>Breaking the first rule causes us some grief sometimes with code that
simply
>will not instrument, but this is pretty rare and usually has to do with the
>differences between the cross-compiler's understanding of Ada and Apex's
>understanding of Ada.  Breaking the second rule is what hurts us the most
by
>not having the test cases automated which takes the most time to create
each
>test case and test by hand all of our decisions and conditions in our code
>then giving Testmate the results.  If you have an automated way of runing
>through your test cases with scripts and collecting coverage, then you will
>really like this tool.  Where can I get information on the other tools you
>mentioned?  Thank you.
>
>Brian Collins
>brian.collins@www.mebbs.com
>bjcollin@rockwell.collins.com
>http://www.math.swt.edu/~rambo
>
>David Akister wrote in message <374667df@news.compd.com>...
>>I would like to know if anyone  can share experience of using the Rational
>>Testmate tool with Rational Apex on UNIX. Of particular interest is the
>>management of code units, time taken developing and modifying test-sets
and
>>flexibility of the system.
>>
>>Oh, and before I forget how does the system compare with other test suites
>>for ADA testing such as ADATEST and LDRA (for coverage metrics) ?
>>
>>Any help is much appreciated
>>Regards
>>
>>Dave Akister
>>Senior Software Engineer
>>Wellgate Technology Ltd
>>
>>
>>
>>
>
>

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Robert Dewar]

In article <7i9du6$1a2$1@lure.pipex.net>,
  "David Akister" <wellgate@dial.pipex.com> wrote:

> With the need for
> validated compilers for safety-critical use this does make
> life difficult.

I find this a bit worrying for two reasons.

1) I worry that people read more into validation than is
appropriate. Validation is useful as one of many tests for
compiler quality, but that is all.

2) Safety critical applications are quite likely to be using
an Ada subset like CSMART, which cannot possibly be validated
in any case.

If the requirement for using a base compiler technology that
is validated is just one of many requirements, then that is
fine, but I have several times run into people who seem to
think that this is

a) a vital requirement (it is not)

b) some kind of assurance of safety (it is not)

With regard to Ada testing, a lot depends on what you are trying
to achieve. Certainly you should be able to find coverage tools
on the actual target you are running on, and there are also a
number of test generation tools that are applicable to many
different target environments.

Robert Dewar
Ada Core Technologies


--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Brian Collins]

Robert,

Depends on your definition of validation.  In the Avionics wold,
compiler validation isn't such a big issue.  The big issue is
verification, not validation.  According to the FAA in numerous
volumes of references,which I shall not name here for the sake
of the groups sleeping habits, verification is very important.  In
any given piece of equipment in an aircraft, every module and
line of software needs to be verified both functionally and even
structurally.  This has to include all software in a piece of equipment
like a compiler Run-Time System (RTS) and any user written
control software as well.  Having a validated compiler, as I think
you mean the word, is important to say that functionally the compiler
will produce the correct code, but that is only a very minor step.  I am
not familiar with the product you named CSMART, but if it is ever
used in an avionics box then it has been verified both functionally
and structurally.  So in essence you have to have validated products.
All depends on your definition of the terms of course.

Brian Collins
brian.collins@www.mebbs.com
bjcollin@collins.rockwell.com
http://www.math.swt.edu/~rambo




Robert Dewar wrote in message <7ia9uj$30r$1@nnrp1.deja.com>...
>In article <7i9du6$1a2$1@lure.pipex.net>,
>  "David Akister" <wellgate@dial.pipex.com> wrote:
>
>> With the need for
>> validated compilers for safety-critical use this does make
>> life difficult.
>
>I find this a bit worrying for two reasons.
>
>1) I worry that people read more into validation than is
>appropriate. Validation is useful as one of many tests for
>compiler quality, but that is all.
>
>2) Safety critical applications are quite likely to be using
>an Ada subset like CSMART, which cannot possibly be validated
>in any case.
>
>If the requirement for using a base compiler technology that
>is validated is just one of many requirements, then that is
>fine, but I have several times run into people who seem to
>think that this is
>
>a) a vital requirement (it is not)
>
>b) some kind of assurance of safety (it is not)
>
>With regard to Ada testing, a lot depends on what you are trying
>to achieve. Certainly you should be able to find coverage tools
>on the actual target you are running on, and there are also a
>number of test generation tools that are applicable to many
>different target environments.
>
>Robert Dewar
>Ada Core Technologies
>
>
>--== Sent via Deja.com http://www.deja.com/ ==--
>---Share what you know. Learn what you don't.---

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Robert Dewar]

In article <7id9ss$r41$1@ins8.netins.net>,
  "Brian Collins" <bjcollin@hotmail.com> wrote:
> Robert,
>
> Having a validated compiler, as I think
> you mean the word, is important to say that functionally the
> compiler will produce the correct code, but that is only a
> very minor step.

I know perfectly well what is involved in verification of
safety-critical code, but I am afraid you miss my point.

Validation of the compiler does NOT "say" that "functionally
the compiler will produce the correct code". Validation of
a compiler is neither necessary nor sufficient to make such
a guarantee.

That was my point, I am always worried that people read more
into validation than is there, and the folks who REALLY know
what verification is about are all too quick to assume that
precisely that validation means that you have this assurance.
it does not, it merely means that a set of tests have passed.
These tests are *entirely* black box tests, so nothing like
coverage or branch testing of the compiler is assured by the
fact that it passes the ACVC tests.

Robert Dewar

P.S. Here at Ada Core Technologies, we are indeed quite aware
of what is involved in certifying and verifying code. That is
why our approach to safety-critical systems is to provide GNORT,
a version of GNAT with absolutely no run-time at all (which
means we don't have to certify our run-time!)



--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Ehud Lamm]

Hi

1) I for one would be happy to see some references about verification in
the avionics world. Verification is a pet subject. You can email me if you
want to answer and think the group will be bored to death.

2) I am not sure what exactly do you mean by "verifaction " "functionally"
and "strcturaly". You surely don't mean "verifiaction" in the sense I am
usually used to i.e., mathematically proven to abide to a formal
specification (ala Z, temporal logic etc.) Do you?

Ehud Lamm     mslamm@pluto.mscc.huji.ac.il

On Mon, 24 May 1999, Brian Collins wrote:

> Robert,
> 
> Depends on your definition of validation.  In the Avionics wold,
> compiler validation isn't such a big issue.  The big issue is
> verification, not validation.  According to the FAA in numerous
> volumes of references,which I shall not name here for the sake
> of the groups sleeping habits, verification is very important.  In
> any given piece of equipment in an aircraft, every module and
> line of software needs to be verified both functionally and even
> structurally.  This has to include all software in a piece of equipment
> like a compiler Run-Time System (RTS) and any user written
> control software as well.  Having a validated compiler, as I think
> you mean the word, is important to say that functionally the compiler
> will produce the correct code, but that is only a very minor step.  I am
> not familiar with the product you named CSMART, but if it is ever
> used in an avionics box then it has been verified both functionally
> and structurally.  So in essence you have to have validated products.
> All depends on your definition of the terms of course.
> 
> Brian Collins
> brian.collins@www.mebbs.com
> bjcollin@collins.rockwell.com
> http://www.math.swt.edu/~rambo
> 
> 
> 
> 
> Robert Dewar wrote in message <7ia9uj$30r$1@nnrp1.deja.com>...
> >In article <7i9du6$1a2$1@lure.pipex.net>,
> >  "David Akister" <wellgate@dial.pipex.com> wrote:
> >
> >> With the need for
> >> validated compilers for safety-critical use this does make
> >> life difficult.
> >
> >I find this a bit worrying for two reasons.
> >
> >1) I worry that people read more into validation than is
> >appropriate. Validation is useful as one of many tests for
> >compiler quality, but that is all.
> >
> >2) Safety critical applications are quite likely to be using
> >an Ada subset like CSMART, which cannot possiblybe validated
> >in any case.
> >
> >If the requirement for using a base compiler technology that
> >is validated is just one of many requirements, then that is
> >fine, but I have several times run into people who seem to
> >think that this is
> >
> >a) a vital requirement (it is not)
> >
> >b) some kind of assurance of safety (it is not)
> >
> >With regard to Ada testing, a lot depends on what you are trying
> >to achieve. Certainly you should be able to find coverage tools
> >on the actual target you are running on,and there are also a
> >number of test generation tools that are applicable to many
> >different target environments.
> >
> >Robert Dewar
> >Ada Core Technologies
> >
> >
> >--== Sent via Deja.com http://www.deja.com/ ==--
> >---Share what you know. Learn what you don't.---
> 
> 
> 
> 

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[Larry Kilgallen]

In article <7ia9uj$30r$1@nnrp1.deja.com>, Robert Dewar <dewar@gnat.com> writes:

> With regard to Ada testing, a lot depends on what you are trying
> to achieve. Certainly you should be able to find coverage tools
> on the actual target you are running on, and there are also a
> number of test generation tools that are applicable to many
> different target environments.

And if you cannot find tools for your platform, you should consider
building them.

The errors Ada will catch is only a subset of the errors humans will make.

Larry Kilgallen

Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ?

[ian_gilchrist]

In article <7i99ls$lv$1@ins8.netins.net>,
  "Brian Collins" <bjcollin@hotmail.com> wrote:
> Dave,
>
> I have used the Testmate MCDC tool on many
verification projects involving
> embedded avionics software.  If your code uses
the Rational Ada compiler and
> the workstation you are using it on is both the
host and target environment,
> then everything is very cool and this is a very
slick tool to use.  When you
> deviate from those two rules the tool continues
to operate, but it will take
> a lot more  effort to keep it going.  You can
still keep the test cases in
> Testmate and keep track of your verification
effort, but it becomes much
> less automated.  In our environment we have to
deviate from both rules by
> first using a non-Apex cross-compiler via RCI
and by second having our
> target environment in our actual hardware box
and not on the workstation.
> Breaking the first rule causes us some grief
sometimes with code that simply
> will not instrument, but this is pretty rare
and usually has to do with the
> differences between the cross-compiler's
understanding of Ada and Apex's
> understanding of Ada.  Breaking the second rule
is what hurts us the most by
> not having the test cases automated which takes
the most time to create each
> test case and test by hand all of our decisions
and conditions in our code
> then giving Testmate the results.  If you have
an automated way of runing
> through your test cases with scripts and
collecting coverage, then you will
> really like this tool.  Where can I get
information on the other tools you
> mentioned?  Thank you.

You can find out more about AdaTEST by looking at
the IPL Web page (www.iplbath.com/tools). AdaTEST
operates on both host and target environments and
also provides MC/DC coverage analysis.

Ian Gilchrist


--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---