* RATIONAL TESTMATE for ADA Testing - Any experiences to share ? @ 1999-05-22 0:00 David Akister 1999-05-22 0:00 ` Ray Blaak 1999-05-23 0:00 ` Brian Collins 0 siblings, 2 replies; 11+ messages in thread From: David Akister @ 1999-05-22 0:00 UTC (permalink / raw) I would like to know if anyone can share experience of using the Rational Testmate tool with Rational Apex on UNIX. Of particular interest is the management of code units, time taken developing and modifying test-sets and flexibility of the system. Oh, and before I forget how does the system compare with other test suites for ADA testing such as ADATEST and LDRA (for coverage metrics) ? Any help is much appreciated Regards Dave Akister Senior Software Engineer Wellgate Technology Ltd ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-22 0:00 RATIONAL TESTMATE for ADA Testing - Any experiences to share ? David Akister @ 1999-05-22 0:00 ` Ray Blaak 1999-05-23 0:00 ` Julius Lancer 1999-05-23 0:00 ` Brian Collins 1 sibling, 1 reply; 11+ messages in thread From: Ray Blaak @ 1999-05-22 0:00 UTC (permalink / raw) "David Akister" <dakister@compd.com> writes: > I would like to know if anyone can share experience of using the Rational > Testmate tool with Rational Apex on UNIX. Of particular interest is the > management of code units, time taken developing and modifying test-sets and > flexibility of the system. I have used it on a large air traffic control project, and found it quite useful for unit testing within a subsystem. One can test against expected outputs, or write general test drivers that set pass/fail explicitly. If one controls the test outputs under CMVC, then one has a nice historical record of how tests pass/fail. Time taken developing tests is rather dependent on the unit being tested, rather than testmate itself (basically, testmate doesn't really get in the way). Testing code should be considered as official software units that get carefully developed with future maintenance and enhancements in mind, reviewed as part of the unit, and need to be budgeted accordingly. (The biggest mistake I saw was to see testing code written as if it was throw-away code -- it never was, and all too often cut/paste/spaghetti code need to be enhanced to test new unit changes). The best feature of testmate that I liked was the code coverage analysis, which highlights those code regions not executed in a test. Basically, testmate is a good tool, but one also needs a good development process to enforce 1) that things are tested at all, 2) how they are tested (i.e. what information needs to be in the test output, and what is deemed sufficient coverage, etc.), and 3) what kinds of test information needs to be saved. > Oh, and before I forget how does the system compare with other test suites > for ADA testing such as ADATEST and LDRA (for coverage metrics) ? I have not used them. -- Cheers, The Rhythm is around me, The Rhythm has control. Ray Blaak The Rhythm is inside me, blaak@infomatch.com The Rhythm has my soul. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-22 0:00 ` Ray Blaak @ 1999-05-23 0:00 ` Julius Lancer 0 siblings, 0 replies; 11+ messages in thread From: Julius Lancer @ 1999-05-23 0:00 UTC (permalink / raw) In article <m3aeuwsle6.fsf@ns32.infomatch.bc.ca>, Ray Blaak <blaak@infomatch.com> wrote: >Basically, testmate is a good tool, but one also needs a good development >process to enforce 1) that things are tested at all, 2) how they are tested >(i.e. what information needs to be in the test output, and what is deemed >sufficient coverage, etc.), and 3) what kinds of test information needs to be >saved. Sounds like some mouse needs to tie the bell around the cat's neck. John - N8086N Big brother is watching. Disable cookies in your web browser. ------------------------------------------- Wise man says "Never use a bank with the initials F. U." ------------------------------------------- Are you interested in a professional society or guild for programmers? Want to fight section 1706? See www.programmersguild.org Newsgroup: us.issues.occupations.computer-programmers EMail Address: _m-i-a-n-o_@_c_o_l_o_s_s_e_u_m_b_u_i_l_d_e_r_s._c_o_m_ ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-22 0:00 RATIONAL TESTMATE for ADA Testing - Any experiences to share ? David Akister 1999-05-22 0:00 ` Ray Blaak @ 1999-05-23 0:00 ` Brian Collins 1999-05-23 0:00 ` David Akister 1999-05-26 0:00 ` ian_gilchrist 1 sibling, 2 replies; 11+ messages in thread From: Brian Collins @ 1999-05-23 0:00 UTC (permalink / raw) Dave, I have used the Testmate MCDC tool on many verification projects involving embedded avionics software. If your code uses the Rational Ada compiler and the workstation you are using it on is both the host and target environment, then everything is very cool and this is a very slick tool to use. When you deviate from those two rules the tool continues to operate, but it will take a lot more effort to keep it going. You can still keep the test cases in Testmate and keep track of your verification effort, but it becomes much less automated. In our environment we have to deviate from both rules by first using a non-Apex cross-compiler via RCI and by second having our target environment in our actual hardware box and not on the workstation. Breaking the first rule causes us some grief sometimes with code that simply will not instrument, but this is pretty rare and usually has to do with the differences between the cross-compiler's understanding of Ada and Apex's understanding of Ada. Breaking the second rule is what hurts us the most by not having the test cases automated which takes the most time to create each test case and test by hand all of our decisions and conditions in our code then giving Testmate the results. If you have an automated way of runing through your test cases with scripts and collecting coverage, then you will really like this tool. Where can I get information on the other tools you mentioned? Thank you. Brian Collins brian.collins@www.mebbs.com bjcollin@rockwell.collins.com http://www.math.swt.edu/~rambo David Akister wrote in message <374667df@news.compd.com>... >I would like to know if anyone can share experience of using the Rational >Testmate tool with Rational Apex on UNIX. Of particular interest is the >management of code units, time taken developing and modifying test-sets and >flexibility of the system. > >Oh, and before I forget how does the system compare with other test suites >for ADA testing such as ADATEST and LDRA (for coverage metrics) ? > >Any help is much appreciated >Regards > >Dave Akister >Senior Software Engineer >Wellgate Technology Ltd > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-23 0:00 ` Brian Collins @ 1999-05-23 0:00 ` David Akister 1999-05-24 0:00 ` Robert Dewar 1999-05-26 0:00 ` ian_gilchrist 1 sibling, 1 reply; 11+ messages in thread From: David Akister @ 1999-05-23 0:00 UTC (permalink / raw) Brian, Thanks for your response, I think you hit the nail on the head certainly with my impressions of the TESTMATE suite. It seems well integrated with the Apex environment, but the need to cross-compile on a non-Rational target compiler means that a lot of the advantage is lost. With the need for validated compilers for safety-critical use this does make life difficult. Additionally one of the main selling points appears to be the ability to provide code coverage listings based on 'Persistant Intermediate representations' ie object code. This is is good stuff if you're using the prescribed compiler.... Generally though the demonstration that I was party to was very impressive, and generation of simple test cases and stubs could be usefully auto-generated. Test-case replicators allow test-lists to be easily produced from the test case template. All good stuff in my opinion... a lot of time is usually spent manually reproducing test cases with varying parameters and managing them.This makes things pretty easy. Overall then, assuming that the end customer is happy with the choice of cross-compiler, it has my recommendation (for what its worth !) LDRA stands for Liverpool Data Research Associates and they produce a very good suite of tools for static analysis and dynamic code coverage. This generates some very useful complexity and knotting reports but in my experience it's used primarily for code coverage, with alternate forms of static analysis tools are used for assessment and possible proving. examples of these are Spark (excellent) and MALPAS (good output but heavy going). If you're interested, I'll try and dig out some contact numbers. Regards David Akister Brian Collins wrote in message <7i99ls$lv$1@ins8.netins.net>... >Dave, > >I have used the Testmate MCDC tool on many verification projects involving >embedded avionics software. If your code uses the Rational Ada compiler and >the workstation you are using it on is both the host and target environment, >then everything is very cool and this is a very slick tool to use. When you >deviate from those two rules the tool continues to operate, but it will take >a lot more effort to keep it going. You can still keep the test cases in >Testmate and keep track of your verification effort, but it becomes much >less automated. In our environment we have to deviate from both rules by >first using a non-Apex cross-compiler via RCI and by second having our >target environment in our actual hardware box and not on the workstation. >Breaking the first rule causes us some grief sometimes with code that simply >will not instrument, but this is pretty rare and usually has to do with the >differences between the cross-compiler's understanding of Ada and Apex's >understanding of Ada. Breaking the second rule is what hurts us the most by >not having the test cases automated which takes the most time to create each >test case and test by hand all of our decisions and conditions in our code >then giving Testmate the results. If you have an automated way of runing >through your test cases with scripts and collecting coverage, then you will >really like this tool. Where can I get information on the other tools you >mentioned? Thank you. > >Brian Collins >brian.collins@www.mebbs.com >bjcollin@rockwell.collins.com >http://www.math.swt.edu/~rambo > >David Akister wrote in message <374667df@news.compd.com>... >>I would like to know if anyone can share experience of using the Rational >>Testmate tool with Rational Apex on UNIX. Of particular interest is the >>management of code units, time taken developing and modifying test-sets and >>flexibility of the system. >> >>Oh, and before I forget how does the system compare with other test suites >>for ADA testing such as ADATEST and LDRA (for coverage metrics) ? >> >>Any help is much appreciated >>Regards >> >>Dave Akister >>Senior Software Engineer >>Wellgate Technology Ltd >> >> >> >> > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-23 0:00 ` David Akister @ 1999-05-24 0:00 ` Robert Dewar 1999-05-24 0:00 ` Brian Collins 1999-05-24 0:00 ` Larry Kilgallen 0 siblings, 2 replies; 11+ messages in thread From: Robert Dewar @ 1999-05-24 0:00 UTC (permalink / raw) In article <7i9du6$1a2$1@lure.pipex.net>, "David Akister" <wellgate@dial.pipex.com> wrote: > With the need for > validated compilers for safety-critical use this does make > life difficult. I find this a bit worrying for two reasons. 1) I worry that people read more into validation than is appropriate. Validation is useful as one of many tests for compiler quality, but that is all. 2) Safety critical applications are quite likely to be using an Ada subset like CSMART, which cannot possibly be validated in any case. If the requirement for using a base compiler technology that is validated is just one of many requirements, then that is fine, but I have several times run into people who seem to think that this is a) a vital requirement (it is not) b) some kind of assurance of safety (it is not) With regard to Ada testing, a lot depends on what you are trying to achieve. Certainly you should be able to find coverage tools on the actual target you are running on, and there are also a number of test generation tools that are applicable to many different target environments. Robert Dewar Ada Core Technologies --== Sent via Deja.com http://www.deja.com/ ==-- ---Share what you know. Learn what you don't.--- ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-24 0:00 ` Robert Dewar @ 1999-05-24 0:00 ` Brian Collins 1999-05-25 0:00 ` Robert Dewar 1999-05-26 0:00 ` Ehud Lamm 1999-05-24 0:00 ` Larry Kilgallen 1 sibling, 2 replies; 11+ messages in thread From: Brian Collins @ 1999-05-24 0:00 UTC (permalink / raw) Robert, Depends on your definition of validation. In the Avionics wold, compiler validation isn't such a big issue. The big issue is verification, not validation. According to the FAA in numerous volumes of references,which I shall not name here for the sake of the groups sleeping habits, verification is very important. In any given piece of equipment in an aircraft, every module and line of software needs to be verified both functionally and even structurally. This has to include all software in a piece of equipment like a compiler Run-Time System (RTS) and any user written control software as well. Having a validated compiler, as I think you mean the word, is important to say that functionally the compiler will produce the correct code, but that is only a very minor step. I am not familiar with the product you named CSMART, but if it is ever used in an avionics box then it has been verified both functionally and structurally. So in essence you have to have validated products. All depends on your definition of the terms of course. Brian Collins brian.collins@www.mebbs.com bjcollin@collins.rockwell.com http://www.math.swt.edu/~rambo Robert Dewar wrote in message <7ia9uj$30r$1@nnrp1.deja.com>... >In article <7i9du6$1a2$1@lure.pipex.net>, > "David Akister" <wellgate@dial.pipex.com> wrote: > >> With the need for >> validated compilers for safety-critical use this does make >> life difficult. > >I find this a bit worrying for two reasons. > >1) I worry that people read more into validation than is >appropriate. Validation is useful as one of many tests for >compiler quality, but that is all. > >2) Safety critical applications are quite likely to be using >an Ada subset like CSMART, which cannot possibly be validated >in any case. > >If the requirement for using a base compiler technology that >is validated is just one of many requirements, then that is >fine, but I have several times run into people who seem to >think that this is > >a) a vital requirement (it is not) > >b) some kind of assurance of safety (it is not) > >With regard to Ada testing, a lot depends on what you are trying >to achieve. Certainly you should be able to find coverage tools >on the actual target you are running on, and there are also a >number of test generation tools that are applicable to many >different target environments. > >Robert Dewar >Ada Core Technologies > > >--== Sent via Deja.com http://www.deja.com/ ==-- >---Share what you know. Learn what you don't.--- ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-24 0:00 ` Brian Collins @ 1999-05-25 0:00 ` Robert Dewar 1999-05-26 0:00 ` Ehud Lamm 1 sibling, 0 replies; 11+ messages in thread From: Robert Dewar @ 1999-05-25 0:00 UTC (permalink / raw) In article <7id9ss$r41$1@ins8.netins.net>, "Brian Collins" <bjcollin@hotmail.com> wrote: > Robert, > > Having a validated compiler, as I think > you mean the word, is important to say that functionally the > compiler will produce the correct code, but that is only a > very minor step. I know perfectly well what is involved in verification of safety-critical code, but I am afraid you miss my point. Validation of the compiler does NOT "say" that "functionally the compiler will produce the correct code". Validation of a compiler is neither necessary nor sufficient to make such a guarantee. That was my point, I am always worried that people read more into validation than is there, and the folks who REALLY know what verification is about are all too quick to assume that precisely that validation means that you have this assurance. it does not, it merely means that a set of tests have passed. These tests are *entirely* black box tests, so nothing like coverage or branch testing of the compiler is assured by the fact that it passes the ACVC tests. Robert Dewar P.S. Here at Ada Core Technologies, we are indeed quite aware of what is involved in certifying and verifying code. That is why our approach to safety-critical systems is to provide GNORT, a version of GNAT with absolutely no run-time at all (which means we don't have to certify our run-time!) --== Sent via Deja.com http://www.deja.com/ ==-- ---Share what you know. Learn what you don't.--- ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-24 0:00 ` Brian Collins 1999-05-25 0:00 ` Robert Dewar @ 1999-05-26 0:00 ` Ehud Lamm 1 sibling, 0 replies; 11+ messages in thread From: Ehud Lamm @ 1999-05-26 0:00 UTC (permalink / raw) Hi 1) I for one would be happy to see some references about verification in the avionics world. Verification is a pet subject. You can email me if you want to answer and think the group will be bored to death. 2) I am not sure what exactly do you mean by "verifaction " "functionally" and "strcturaly". You surely don't mean "verifiaction" in the sense I am usually used to i.e., mathematically proven to abide to a formal specification (ala Z, temporal logic etc.) Do you? Ehud Lamm mslamm@pluto.mscc.huji.ac.il On Mon, 24 May 1999, Brian Collins wrote: > Robert, > > Depends on your definition of validation. In the Avionics wold, > compiler validation isn't such a big issue. The big issue is > verification, not validation. According to the FAA in numerous > volumes of references,which I shall not name here for the sake > of the groups sleeping habits, verification is very important. In > any given piece of equipment in an aircraft, every module and > line of software needs to be verified both functionally and even > structurally. This has to include all software in a piece of equipment > like a compiler Run-Time System (RTS) and any user written > control software as well. Having a validated compiler, as I think > you mean the word, is important to say that functionally the compiler > will produce the correct code, but that is only a very minor step. I am > not familiar with the product you named CSMART, but if it is ever > used in an avionics box then it has been verified both functionally > and structurally. So in essence you have to have validated products. > All depends on your definition of the terms of course. > > Brian Collins > brian.collins@www.mebbs.com > bjcollin@collins.rockwell.com > http://www.math.swt.edu/~rambo > > > > > Robert Dewar wrote in message <7ia9uj$30r$1@nnrp1.deja.com>... > >In article <7i9du6$1a2$1@lure.pipex.net>, > > "David Akister" <wellgate@dial.pipex.com> wrote: > > > >> With the need for > >> validated compilers for safety-critical use this does make > >> life difficult. > > > >I find this a bit worrying for two reasons. > > > >1) I worry that people read more into validation than is > >appropriate. Validation is useful as one of many tests for > >compiler quality, but that is all. > > > >2) Safety critical applications are quite likely to be using > >an Ada subset like CSMART, which cannot possiblybe validated > >in any case. > > > >If the requirement for using a base compiler technology that > >is validated is just one of many requirements, then that is > >fine, but I have several times run into people who seem to > >think that this is > > > >a) a vital requirement (it is not) > > > >b) some kind of assurance of safety (it is not) > > > >With regard to Ada testing, a lot depends on what you are trying > >to achieve. Certainly you should be able to find coverage tools > >on the actual target you are running on,and there are also a > >number of test generation tools that are applicable to many > >different target environments. > > > >Robert Dewar > >Ada Core Technologies > > > > > >--== Sent via Deja.com http://www.deja.com/ ==-- > >---Share what you know. Learn what you don't.--- > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-24 0:00 ` Robert Dewar 1999-05-24 0:00 ` Brian Collins @ 1999-05-24 0:00 ` Larry Kilgallen 1 sibling, 0 replies; 11+ messages in thread From: Larry Kilgallen @ 1999-05-24 0:00 UTC (permalink / raw) In article <7ia9uj$30r$1@nnrp1.deja.com>, Robert Dewar <dewar@gnat.com> writes: > With regard to Ada testing, a lot depends on what you are trying > to achieve. Certainly you should be able to find coverage tools > on the actual target you are running on, and there are also a > number of test generation tools that are applicable to many > different target environments. And if you cannot find tools for your platform, you should consider building them. The errors Ada will catch is only a subset of the errors humans will make. Larry Kilgallen ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RATIONAL TESTMATE for ADA Testing - Any experiences to share ? 1999-05-23 0:00 ` Brian Collins 1999-05-23 0:00 ` David Akister @ 1999-05-26 0:00 ` ian_gilchrist 1 sibling, 0 replies; 11+ messages in thread From: ian_gilchrist @ 1999-05-26 0:00 UTC (permalink / raw) In article <7i99ls$lv$1@ins8.netins.net>, "Brian Collins" <bjcollin@hotmail.com> wrote: > Dave, > > I have used the Testmate MCDC tool on many verification projects involving > embedded avionics software. If your code uses the Rational Ada compiler and > the workstation you are using it on is both the host and target environment, > then everything is very cool and this is a very slick tool to use. When you > deviate from those two rules the tool continues to operate, but it will take > a lot more effort to keep it going. You can still keep the test cases in > Testmate and keep track of your verification effort, but it becomes much > less automated. In our environment we have to deviate from both rules by > first using a non-Apex cross-compiler via RCI and by second having our > target environment in our actual hardware box and not on the workstation. > Breaking the first rule causes us some grief sometimes with code that simply > will not instrument, but this is pretty rare and usually has to do with the > differences between the cross-compiler's understanding of Ada and Apex's > understanding of Ada. Breaking the second rule is what hurts us the most by > not having the test cases automated which takes the most time to create each > test case and test by hand all of our decisions and conditions in our code > then giving Testmate the results. If you have an automated way of runing > through your test cases with scripts and collecting coverage, then you will > really like this tool. Where can I get information on the other tools you > mentioned? Thank you. You can find out more about AdaTEST by looking at the IPL Web page (www.iplbath.com/tools). AdaTEST operates on both host and target environments and also provides MC/DC coverage analysis. Ian Gilchrist --== Sent via Deja.com http://www.deja.com/ ==-- ---Share what you know. Learn what you don't.--- ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~1999-05-26 0:00 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 1999-05-22 0:00 RATIONAL TESTMATE for ADA Testing - Any experiences to share ? David Akister 1999-05-22 0:00 ` Ray Blaak 1999-05-23 0:00 ` Julius Lancer 1999-05-23 0:00 ` Brian Collins 1999-05-23 0:00 ` David Akister 1999-05-24 0:00 ` Robert Dewar 1999-05-24 0:00 ` Brian Collins 1999-05-25 0:00 ` Robert Dewar 1999-05-26 0:00 ` Ehud Lamm 1999-05-24 0:00 ` Larry Kilgallen 1999-05-26 0:00 ` ian_gilchrist
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox