Is Aunit helpful?

Is Aunit helpful?

[Ada novice]

Hi, being a beginner in Ada and in software engineering, I came across
a textbook mentioning aunit for writing test drivers. I write quite
simple Ada programs and was wondering if it's worth to learn aunit and
use it even on simple programs. I understand that it can be a time-
saver.

Thanks

YC

Re: Is Aunit helpful?

[Stephen Leake]

Ada novice <posts@gmx.us> writes:

> Hi, being a beginner in Ada and in software engineering, I came across
> a textbook mentioning aunit for writing test drivers. I write quite
> simple Ada programs and was wondering if it's worth to learn aunit and
> use it even on simple programs. I understand that it can be a time-
> saver.

Yes, it is a good way to organize your testing.

It will not save time initially, but it will definitely save time in the
long run.

-- 
-- Stephe

Re: Is Aunit helpful?

[Ada novice]

On Aug 5, 2:07 pm, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:

> Yes, it is a good way to organize your testing.

I tried to look at the examples supplied in the aunit manual and they
aren't so straightforward to me. Are there other sources (on the web?)
that show how to use this testing tool? Or maybe I should just print
out the manual and read it carefully.

>
> It will not save time initially, but it will definitely save time in the
> long run.

It's definitely better in the long run to have good organization in
the codes and do things systematically in a proper way using a testing
tool such as aunit.


Thanks
YC

Re: Is Aunit helpful?

[Tero Koskinen]

On Thu, 5 Aug 2010 10:29:05 -0700 (PDT) Ada novice wrote:

> On Aug 5, 2:07 pm, Stephen Leake <stephen_le...@stephe-leake.org>
> wrote:
> 
> > Yes, it is a good way to organize your testing.
> 
> I tried to look at the examples supplied in the aunit manual and they
> aren't so straightforward to me. Are there other sources (on the web?)
> that show how to use this testing tool? Or maybe I should just print
> out the manual and read it carefully.

AdaCommons wiki has a small article and an example:
http://www.adacommons.org/AUnit

http://www.adacommons.org/AUnit_Calculator_Example

The example is for AUnit 1.x. It most likely doesn't work out of the box
with AUnit 2.x or 3.x, since every AUnit major version has somewhat 
different API.

> Thanks
> YC


-- 
Tero Koskinen - http://iki.fi/tero.koskinen/

Re: Is Aunit helpful?

[Ada novice]

On Aug 5, 8:18 pm, Tero Koskinen <tero.koski...@iki.fi> wrote:

> AdaCommons wiki has a small article and an example:http://www.adacommons.org/AUnit
>
> http://www.adacommons.org/AUnit_Calculator_Example
>
> The example is for AUnit 1.x. It most likely doesn't work out of the box
> with AUnit 2.x or 3.x, since every AUnit major version has somewhat
> different API.
>

Thanks. I did find this example earlier but the commands given there:

gnatmake -Pcalc
./harness -v

doesn't seem right. Though gnatmake -Pcalc is correct and works, I
don't understand what the additional "./harness " is for.

YC

Re: Is Aunit helpful?

[Simon Wright]

Ada novice <posts@gmx.us> writes:

> On Aug 5, 8:18 pm, Tero Koskinen <tero.koski...@iki.fi> wrote:
>
>> AdaCommons wiki has a small article and an
>> example:http://www.adacommons.org/AUnit
>>
>> http://www.adacommons.org/AUnit_Calculator_Example
>>
>> The example is for AUnit 1.x. It most likely doesn't work out of the
>> box with AUnit 2.x or 3.x, since every AUnit major version has
>> somewhat different API.
>>
>
> Thanks. I did find this example earlier but the commands given there:
>
> gnatmake -Pcalc
> ./harness -v
>
> doesn't seem right. Though gnatmake -Pcalc is correct and works, I
> don't understand what the additional "./harness " is for.

To run the test!

harness is the executable (harness.exe on Windows). On Unix systems it's
normal not to include the current directory in the path; so to run the
program you have to say where it is, ".", and the directory separator is
a plain slash, not that nasty backslash.

On Windows you probably just need to say "harness -v".

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Thu, 05 Aug 2010 22:39:00 +0200, Ada novice <posts@gmx.us> a écrit:
> gnatmake -Pcalc
> ./harness -v

It seems there is another GNAT 2010 trick here. In project files (*.gpr  
files), this may be required to turn all "-gnatwa.Xe" into "-gnatwa.X".  
"-gnatwa.Xe" tells GPRBuild to treat warnings as error, and there are  
warnings like
“math-test.adb:13:07: warning: call to obsolescent procedure "Assert"  
declared at aunit-assertions.ads:113”
which occurs and would otherwise makes the compilation failed (as this end  
up to be treated as error).

About AUnit: just seen about what it is, how it is set up and how it  
works. Seems a question is still pending : “how to be sure the test cover  
all relevant case ?”. I do not see a way to be sure testing cover all  
cases. That is the main limitation of this kind of approach. May be this  
is nice for regression tests and tests cases revealed during application's  
life time... however, I don't believe this can be used too much soon in a  
development stage. Pretty sure this is best used just after real use or  
deployment has been started.


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Is Aunit helpful?

[Simon Wright]

"Yannick Duchêne (Hibou57)" <yannick_duchene@yahoo.fr> writes:

> About AUnit: just seen about what it is, how it is set up and how it
> works. Seems a question is still pending : “how to be sure the test
> cover all relevant case ?”. I do not see a way to be sure testing
> cover all cases. That is the main limitation of this kind of
> approach. May be this is nice for regression tests and tests cases
> revealed during application's life time... however, I don't believe
> this can be used too much soon in a development stage. Pretty sure
> this is best used just after real use or deployment has been started.

gcov
(http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gnat_ugn_unw/Code-Coverage-of-Ada-Programs-using-gcov.html#Code-Coverage-of-Ada-Programs-using-gcov)
will help to tell which bits of code your tests haven't exercised at
all. And it's free.

For more capable test coverage tools you'll probably have to pay.

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

> gcov
> (http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gnat_ugn_unw/Code-Coverage-of-Ada-Programs-using-gcov.html#Code-Coverage-of-Ada-Programs-using-gcov)
> will help to tell which bits of code your tests haven't exercised at
> all. And it's free.
You are right. Part of the things I use to know about while still never  
tried (there is also an option in GPS project options dialog).

> For more capable test coverage tools you'll probably have to pay.
I would like I could one day (as well I getting a Janus license, I would  
like to one day)... depends on weither or not some one else would pay for  
stuffs I do (another story).


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Is Aunit helpful?

[Stephen Leake]

"Yannick Duchêne (Hibou57)" <yannick_duchene@yahoo.fr> writes:

> About AUnit: just seen about what it is, how it is set up and how it
> works. Seems a question is still pending : “how to be sure the test
> cover  all relevant case ?”. I do not see a way to be sure testing
> cover all  cases. 

Correct, AUnit does not do that. gcov does, although I have not used it
very much. It can be difficult to use the output of gcov.

> That is the main limitation of this kind of approach. 

What alternative approaches provide coverage information?

> May be this is nice for regression tests and tests cases revealed
> during application's life time... however, I don't believe this can be
> used too much soon in a development stage. 

I use it all the time for development. I write the AUnit test and spec
first, then the body.

The test documents how the package is intended to be used, and helps
focus the development process.

-- 
-- Stephe

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Sat, 14 Aug 2010 08:57:04 +0200, Stephen Leake  
<stephen_leake@stephe-leake.org> a écrit:
> The test documents how the package is intended to be used, and helps
> focus the development process.
As you talked about “to document” this makes me think about another topic:  
I will check if there is a way to have merged in package documentation,  
like something “with this argument this should return this or do that”.  
Finally, these test cases match something normative or at least may  
complete this normative part.


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Is Aunit helpful?

[Midoan]

On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> "Yannick Duchêne (Hibou57)" <yannick_duch...@yahoo.fr> writes:
>
> > About AUnit: just seen about what it is, how it is set up and how it
> > works. Seems a question is still pending : “how to be sure the test
> > cover  all relevant case ?”. I do not see a way to be sure testing
> > cover all  cases.
>
> Correct, AUnit does not do that. gcov does, although I have not used it
> very much. It can be difficult to use the output of gcov.
>
> > That is the main limitation of this kind of approach.
>
> What alternative approaches provide coverage information?
>

FYI, note that with Mika (http://www.midoan.com/), the automatic test
data  generator for Ada, it is possible to take in your existing test
cases, check the coverage achieved, and automatically generate missing
test inputs and expected test result (which can be validated to form
new test cases automatically). This facility will be automated soon:
http://www.midoan.com/soon.html

This can be done to achieve branch, decision or MC/DC coverage as
desired.

regards,
the Midoan team

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Sat, 14 Aug 2010 23:33:32 +0200, Midoan <midoan.ses@gmail.com> a écrit:
> This can be done to achieve branch, decision or MC/DC coverage as
> desired.
What is MC/DC please ?

Re: Is Aunit helpful?

[Simon Wright]

"Yannick Duchêne (Hibou57)" <yannick_duchene@yahoo.fr> writes:

> Le Sat, 14 Aug 2010 23:33:32 +0200, Midoan <midoan.ses@gmail.com> a écrit:
>> This can be done to achieve branch, decision or MC/DC coverage as
>> desired.
> What is MC/DC please ?

http://letmegooglethatforyou.com/?q=MC/DC

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Sat, 14 Aug 2010 23:49:37 +0200, Simon Wright <simon@pushface.org> a  
écrit:
>> What is MC/DC please ?
>
> http://letmegooglethatforyou.com/?q=MC/DC
Sorry, though I was talking to a human (joking)

Humans better account and understand things than robots do

Don't bother will see (but not using the monopolistic Google)

Re: Is Aunit helpful?

[Midoan]

On Aug 14, 10:38 pm, Yannick Duchêne (Hibou57)
<yannick_duch...@yahoo.fr> wrote:
> Le Sat, 14 Aug 2010 23:33:32 +0200, Midoan <midoan....@gmail.com> a écrit:> This can be done to achieve branch, decision or MC/DC coverage as
> > desired.
>
> What is MC/DC please ?

MC/DC stands for Modified Condition/Decision Coverage. Informally, it
forces deep testing of conditional expressions as in "IF A and B or C
THEN ..." where several tests would be required for the "A and B or C"
expression whereas branch and decision testing would only require two
(one for a True outcome and one for a False outcome). It is a more
thorough, if somewhat badly defined, coverage criterion often required
for high integrity systems.

regards,
the Midoan Team

Re: Is Aunit helpful?

[Simon Wright]

Midoan <midoan.ses@gmail.com> writes:

> On Aug 14, 10:38 pm, Yannick Duchêne (Hibou57)
> <yannick_duch...@yahoo.fr> wrote:
>> Le Sat, 14 Aug 2010 23:33:32 +0200, Midoan <midoan....@gmail.com> a
écrit:
>> > This can be done to achieve branch, decision or MC/DC coverage as
>> > desired.
>>
>> What is MC/DC please ?
>
> MC/DC stands for Modified Condition/Decision Coverage. Informally, it
> forces deep testing of conditional expressions as in "IF A and B or C
> THEN ..." where several tests would be required for the "A and B or C"
> expression whereas branch and decision testing would only require two
> (one for a True outcome and one for a False outcome). It is a more
> thorough, if somewhat badly defined, coverage criterion often required
> for high integrity systems.

Reverting to the use of gcov: even if you got full gcov coverage, that
wouldn't tell you whether the False outcome had actually
occurred. (well, there are options to check for that -- but, as far as I
can see on a quick check, they aren't very helpful, because they report
on the branches in the object code, which includes (for instance) all
the range checks that always succeed in a well-written Ada program!).

Most often, tools such as Midoan is discussing will involve
automatically instrumenting the source code, so that they can report at
the appropriate (source) level.

Re: Is Aunit helpful?

[Stephen Leake]

Midoan <midoan.ses@gmail.com> writes:

> On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
> wrote:
>> "Yannick Duchêne (Hibou57)" <yannick_duch...@yahoo.fr> writes:
>>
>> > About AUnit: just seen about what it is, how it is set up and how it
>> > works. Seems a question is still pending : “how to be sure the test
>> > cover  all relevant case ?”. I do not see a way to be sure testing
>> > cover all  cases.
>>
>> Correct, AUnit does not do that. gcov does, although I have not used it
>> very much. It can be difficult to use the output of gcov.
>>
>> > That is the main limitation of this kind of approach.
>>
>> What alternative approaches provide coverage information?
>>
>
> FYI, note that with Mika (http://www.midoan.com/), the automatic test
> data  generator for Ada, it is possible to take in your existing test
> cases, check the coverage achieved, and automatically generate missing
> test inputs and expected test result 

How can a tool possibly generate expected results? If it reads the code,
it can only generate the results that the code _will_ produce. But
that's the opposite of a test; the expected results are what the code
_should_ produce, based on some other spec (not the Ada spec). A testing
process _must_ assume the code is wrong.

If the spec is machine readable, then the tool has a chance. But I see
no mention of machine readable specs on the midoan site.

Hmm. If the Ada spec includes pre/postconditions (Ada 2012), then some
meaningful tests can be generated, but the compiler will already do
that.

> (which can be validated to form new test cases automatically).

validated by what? If that means "reviewed by a human", that might be
ok. But there would be a very strong temptation to say "the tool must be
right". 

> This can be done to achieve branch, decision or MC/DC coverage as
> desired.

Generating a scaffold that gives coverage could be useful, but it must
be completed manually, to ensure correct results.

-- 
-- Stephe

Re: Is Aunit helpful?

[Midoan]

On Aug 15, 1:10 pm, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> Midoan <midoan....@gmail.com> writes:
> > On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
> > wrote:
> >> "Yannick Duchêne (Hibou57)" <yannick_duch...@yahoo.fr> writes:
>
> >> > About AUnit: just seen about what it is, how it is set up and how it
> >> > works. Seems a question is still pending : “how to be sure the test
> >> > cover  all relevant case ?”. I do not see a way to be sure testing
> >> > cover all  cases.
>
> >> Correct, AUnit does not do that. gcov does, although I have not used it
> >> very much. It can be difficult to use the output of gcov.
>
> >> > That is the main limitation of this kind of approach.
>
> >> What alternative approaches provide coverage information?
>
> > FYI, note that with Mika (http://www.midoan.com/), the automatic test
> > data  generator for Ada, it is possible to take in your existing test
> > cases, check the coverage achieved, and automatically generate missing
> > test inputs and expected test result
>
> How can a tool possibly generate expected results? If it reads the code,
> it can only generate the results that the code _will_ produce. But
> that's the opposite of a test; the expected results are what the code
> _should_ produce, based on some other spec (not the Ada spec). A testing
> process _must_ assume the code is wrong.

Yes of course you are right; and that's what we meant. What we meant
was that the code's result must always be validated externally of the
tool. Mika will simply ask the question "given these inputs, are those
expected outputs?". Those outputs, that Mika generates, will be
according to the code under test: it will include all side effects
(i.e. including non local variables changes) ... In our experience,
Mika will flag many side effects not necessarily intended by the
developers, never mind the designers ... If the code's outputs is
however confirmed by the human oracle then it can join a set test
cases suitable for regression testing. If the code is subsequently
changed then Midoan can flag changed behavior and/or complete the test
inputs set; thus being much useful during developement, maintenance
(i.e. regression) testing.

> If the spec is machine readable, then the tool has a chance. But I see
> no mention of machine readable specs on the midoan site.
>
> Hmm. If the Ada spec includes pre/postconditions (Ada 2012), then some
> meaningful tests can be generated, but the compiler will already do
> that.
>
> > (which can be validated to form new test cases automatically).
>
> validated by what? If that means "reviewed by a human", that might be
> ok. But there would be a very strong temptation to say "the tool must be
> right".
>
> > This can be done to achieve branch, decision or MC/DC coverage as
> > desired.
>

We are very open to inquiries about machine readable, commercially
sustainable (preferably!), specification languages as used in
combination with Ada (any specification language based on Ada's syntax
is particularly welcome :-) ). In fact disproving specified behavior
would be a very attractive proposition for us ... But, in our
experience, most specifications (and that includes code generated (!)
specifications used by fashionable model based verification techniques
as provided by Polyspace and such like) only capture a fraction of the
code's behavior; e.g. not all packages' variables are modeled. In
practice, and for example, Ada 2012's post-conditions are unlikely to
be used to capture the full effect of commercial, safety critical,
code ... That, to our eyes, is a major weakness in specification based
verification : the specification may say 'A' but the code may do 'A
and B' ... and trying to model 'B' in the specification typically
leads to  specifications as complex as the code!

So we, as a gist conclusion, at Midoan, still believe in the primacy
of human checked, tool assisted, systematic software testing.

> Generating a scaffold that gives coverage could be useful, but it must
> be completed manually, to ensure correct results.
>

Agreed, indeed.

> --
> -- Stephe

Regards,
The Midoan Team at http://www.midoan.com/

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Sun, 15 Aug 2010 23:47:21 +0200, Midoan <midoan.ses@gmail.com> a écrit:
> In practice, and for example, Ada 2012's post-conditions are unlikely to
> be used to capture the full effect of commercial, safety critical,
> code ... That, to our eyes, is a major weakness in specification based
> verification : the specification may say 'A' but the code may do 'A
> and B' ... and trying to model 'B' in the specification typically
> leads to  specifications as complex as the code!

Well, I like the idea of Ada 2012 pre-post (while still not tried) the Ada  
version of this (I did with the Eiffel one).

I already used this, with Eiffel, and my memory tells me I cannot remember  
any case where this really helped and fixing a bug (you will laugh, but  
even the SmallEiffel compiler source, was containing nothing that here and  
there some seed of toy contract-assertions, very poor use in SmallEiffel  
itself). Either the pre-post was erroneously expressed (and ended into  
signaling violation which was not error) or this was missing some  
important part which could then not be detected. Thus, I agree with you  
about the “weak”. I would even add “possibly wrong” (may be worse than  
weak).

 From there, I would like to get to three points.

1) After my experiences with Eiffel, pre-post could be either erroneous or  
missing something, for a simple reason: this is not statically checked,  
this is only runtime checked... so it is far too much easy to fail at  
design time! (event in a recursive design process, this does not really  
change anything) If this was statically checked, this would be deeply  
different. The weakness here is not due to pre-post concept, but rather to  
static check vs runtime check.

2) Your comment makes me think what you would better enjoy, would be  
reification. You would not have any more troubles with “said A but did A  
and B”, as A and B would have been produced during a process and would be  
under control of this process (there woul be an explicit track forth and  
back to and from this A and B).

3) About point #1, just wanted to say I don't want here to discourage  
people interested in pre-post introduced in Ada 2012, to use it. No-no-no,  
if they enjoy this, this is very good, and this will lead them to a good  
way. Just wanted to say what should not be hidden: this is weak, as this  
is more a step above comments. I was to say “this is just like  
comments”... but honestly, comment are not checked at all, so these  
pre-post may be viewed as “Better Comments”.

-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Is Aunit helpful?

[Randy Brukardt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

"Yannick Duch�ne (Hibou57)" <yannick_duchene@yahoo.fr> wrote in message 
news:op.vhh4slqeule2fv@garhos...
Le Sun, 15 Aug 2010 23:47:21 +0200, Midoan <midoan.ses@gmail.com> a �crit:
...
> 3) About point #1, just wanted to say I don't want here to discourage 
> people interested in pre-post introduced in Ada 2012, to use it. No-no-no, 
> if they enjoy this, this is very good, and this will lead them to a good 
> way. Just wanted to say what should not be hidden: this is weak, as this 
> is more a step above comments. I was to say "this is just like  comments"... 
> but honestly, comment are not checked at all, so these  pre-post may be 
> viewed as "Better Comments".

Not so much "Better Comments", but "automatic pragma Asserts". I don't 
disagree with your point, but Pre/Post are better than pragma Assert in two 
ways:

(1) Pre and Post are automatically added to every call site; it's not 
possible to forget to put in a needed Assert;

(2) Pre and Post are in the specification, and thus are visible to the 
callers. That allows both the Ada compiler and other tools to do static 
analysis. The Ada compiler may be able to prove that the precondition cannot 
fail (and thus eliminate the overhead); other tools may be able to do even a 
better job. That is, well-written preconditions and postconditions can be 
statically proved. (The Ada 2012 language won't require that, because its 
beyond the state-of-the-art today to be able to decide which ones can be 
proved and which ones cannot. But I hope that we can do better in the 
future.)

Pre and Post hopefully will provide a foundation where compilers will make 
at least some of these checks statically.

                           Randy.

Re: Is Aunit helpful?

[Cyrille]

On Aug 17, 4:46 am, "Randy Brukardt" <ra...@rrsoftware.com> wrote:
> fail (and thus eliminate the overhead); other tools may be able to do even a
> better job. That is, well-written preconditions and postconditions can be
> statically proved. (The Ada 2012 language won't require that, because its
> beyond the state-of-the-art today to be able to decide which ones can be
> proved and which ones cannot. But I hope that we can do better in the
> future.)

This is one of the goals of the Hi-Lite project! (http://www.open-
do.org/projects/hi-lite/)

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Tue, 17 Aug 2010 14:37:56 +0200, Cyrille <comar@eu.adacore.com> a écrit:
> This is one of the goals of the Hi-Lite project!
> http://www.open-do.org/projects/hi-lite/
I had a quick look a read a bit the french paper. Seems interesting.
So this is organization whose purpose is to promote formal methods ?
Do they also target solutions ?
Are you, yourself, a member of this team ?

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Tue, 17 Aug 2010 04:46:23 +0200, Randy Brukardt <randy@rrsoftware.com>  
a écrit:
> better job. That is, well-written preconditions and postconditions can be
> statically proved. (The Ada 2012 language won't require that, because its
> beyond the state-of-the-art today to be able to decide which ones can be
> proved and which ones cannot. But I hope that we can do better in the
> future.)
>
> Pre and Post hopefully will provide a foundation where compilers will  
> make
> at least some of these checks statically.
>
>                            Randy.

May be yes, I have underestimated this technique.

I agree with your prospective view

Re: Is Aunit helpful?

[Stephen Leake]

Midoan <midoan.ses@gmail.com> writes:

>> Midoan <midoan....@gmail.com> writes:
>> > On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
>> > wrote:
>>
>> How can a tool possibly generate expected results? If it reads the code,
>> it can only generate the results that the code _will_ produce. But
>> that's the opposite of a test; the expected results are what the code
>> _should_ produce, based on some other spec (not the Ada spec). A testing
>> process _must_ assume the code is wrong.
>
> Yes of course you are right; and that's what we meant. What we meant
> was that the code's result must always be validated externally of the
> tool. 

Ok, good.

> Mika will simply ask the question "given these inputs, are those
> expected outputs?". 

I suggest you emphasize this point on your website.

If a website describes a tool that claims to "automatically generate
tests", I stop reading, since I know that's a bogus claim.

I would consider a tool that "helps identify all required test cases".

-- 
-- Stephe

Re: Is Aunit helpful?

[Midoan]

On Aug 16, 8:59 am, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> Midoan <midoan....@gmail.com> writes:
> >> Midoan <midoan....@gmail.com> writes:
> >> > On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
> >> > wrote:
>
> >> How can a tool possibly generate expected results? If it reads the code,
> >> it can only generate the results that the code _will_ produce. But
> >> that's the opposite of a test; the expected results are what the code
> >> _should_ produce, based on some other spec (not the Ada spec). A testing
> >> process _must_ assume the code is wrong.
>
> > Yes of course you are right; and that's what we meant. What we meant
> > was that the code's result must always be validated externally of the
> > tool.
>
> Ok, good.
>
> > Mika will simply ask the question "given these inputs, are those
> > expected outputs?".
>
> I suggest you emphasize this point on your website.
>
> If a website describes a tool that claims to "automatically generate
> tests", I stop reading, since I know that's a bogus claim.
>
> I would consider a tool that "helps identify all required test cases".
>
> --
> -- Stephe

Stephen,

it depends what you call a test. Formally, Mika does generate test
inputs automatically to achieve full coverage of the code under test:
that is not a bogus claim.

Tests cases on the other hand cannot be generated from code alone: a
test case includes test inputs and expected output according to the
specification or some oracle. Mika can help in this process too but
require human/oracle validation (once per tests) of the outputs.

We not feel that the information on our website is misleading. We are
however in the process of developing better user documentation,
training and support.

Hope this helps.
Regards.
The Midoan Team.

Re: Is Aunit helpful?

[Colin Paul Gloster]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3074 bytes --]

Midoan <midoan.ses@gmail.com> sent on August 16th, 2010:

|-----------------------------------------------------------------------------|
|"On Aug 16, 8:59 am, Stephen Leake <stephen_le...@stephe-leake.org>          |
|wrote:                                                                       |
|[..]                                                                         |
|> > Mika will simply ask the question "given these inputs, are those         |
|> > expected outputs?".                                                      |
|>                                                                            |
|> I suggest you emphasize this point on your website.                        |
|>                                                                            |
|> If a website describes a tool that claims to "automatically generate       |
|> tests", I stop reading, since I know that's a bogus claim.                 |
|>                                                                            |
|> I would consider a tool that "helps identify all required test cases".     |
|>                                                                            |
|> --                                                                         |
|> -- Stephe                                                                  |
|                                                                             |
|Stephen,                                                                     |
|                                                                             |
|it depends what you call a test. Formally, Mika does generate test           |
|inputs automatically to achieve full coverage of the code under test:        |
|that is not a bogus claim.                                                   |
|                                                                             |
|[..]                                                                         |
|                                                                             |
|We not feel that the information on our website is misleading. We are        |
|however in the process of developing better user documentation,              |
|training and support.                                                        |
|                                                                             |
|Hope this helps.                                                             |
|Regards.                                                                     |
|The Midoan Team."                                                            |
|-----------------------------------------------------------------------------|

Your website is not intended to be misleading. You interpret your
website as it was intended to be interpreted. Outsiders (potential
clients) interpret your website as it appears to outsiders, so please
help yourselves by phrasing your website in such a way that potential
clients do not dismiss you.

Good luck,
Paul Colin Gloster

Re: Is Aunit helpful?

[Marco]

On Aug 5, 11:18 am, Tero Koskinen <tero.koski...@iki.fi> wrote:
> On Thu, 5 Aug 2010 10:29:05 -0700 (PDT) Ada novice wrote:
>
> > On Aug 5, 2:07 pm, Stephen Leake <stephen_le...@stephe-leake.org>
> > wrote:
>
> > > Yes, it is a good way to organize your testing.
>
> > I tried to look at the examples supplied in the aunit manual and they
> > aren't so straightforward to me. Are there other sources (on the web?)
> > that show how to use this testing tool? Or maybe I should just print
> > out the manual and read it carefully.
>
> AdaCommons wiki has a small article and an example:http://www.adacommons.org/AUnit
>
> http://www.adacommons.org/AUnit_Calculator_Example

also see
http://libre.adacore.com/libre/tools/aunit/

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Thu, 05 Aug 2010 20:18:20 +0200, Tero Koskinen <tero.koskinen@iki.fi> a  
écrit:
> http://www.adacommons.org/AUnit_Calculator_Example
Seems there is a tiny error there:
“Assert (Calc.Sum (2, 3) = 5, "test 2 +5 = 5");”
instead of
“Assert (Calc.Sum (2, 3) = 5, "test 2 + 3 = 5");”


-- 
There is even better than a pragma Assert: a SPARK --# check.
--# check C and WhoKnowWhat and YouKnowWho;
--# assert Ada;
--  i.e. forget about previous premises which leads to conclusion
--  and start with new conclusion as premise.

Re: Is Aunit helpful?

[Simon Wright]

Ada novice <posts@gmx.us> writes:

> I tried to look at the examples supplied in the aunit manual and they
> aren't so straightforward to me. Are there other sources (on the web?)
> that show how to use this testing tool? Or maybe I should just print
> out the manual and read it carefully.

I found the manual quite hard. And the sample code didn't quite gel for me.

I've used it seriously -- at eg
http://coldframe.cvs.sourceforge.net/viewvc/coldframe/coldframe/Event_Test.test/

The above doesn't have a Software Test Description, so you have to use
your imagination; Harness is the main program, Suite calls up the test
cases, Test_* are the test cases.

But this isn't by any means the only organisation possible.

Re: Is Aunit helpful?

[Ada novice]

On Aug 5, 10:48 pm, Simon Wright <si...@pushface.org> wrote:

> I found the manual quite hard. And the sample code didn't quite gel for me.
>
> I've used it seriously -- at eghttp://coldframe.cvs.sourceforge.net/viewvc/coldframe/coldframe/Event...
>
> The above doesn't have a Software Test Description, so you have to use
> your imagination; Harness is the main program, Suite calls up the test
> cases, Test_* are the test cases.
>


Is coldframe an alternative tool to aunit or does it work with aunit?


Thanks
YC

Re: Is Aunit helpful?

[Simon Wright]

Ada novice <posts@gmx.us> writes:

> Is coldframe an alternative tool to aunit or does it work with aunit?

ColdFrame is a code generator, I used AUnit to test it, this was an
example of an extensive _use_ of AUnit.

Re: Is Aunit helpful?

[Ada novice]

On Aug 6, 6:51 pm, Simon Wright <si...@pushface.org> wrote:

> ColdFrame is a code generator, I used AUnit to test it, this was an
> example of an extensive _use_ of AUnit.

Thanks. There are lots of files within ColdFrame. I need to print some
out and study them.

YC

Re: Is Aunit helpful?

[John McCormick]

On Aug 4, 9:19 am, Ada novice <po...@gmx.us> wrote:
> Hi, being a beginner in Ada and in software engineering, I came across
> a textbook mentioning aunit for writing test drivers. I write quite
> simple Ada programs and was wondering if it's worth to learn aunit and
> use it even on simple programs. I understand that it can be a time-
> saver.
>
> Thanks
>
> YC

You may be interested in a video tutorial on AUnit produced by Daniel
Bigelow (independent consultant, www.bigelow.ch).

http://www.adacore.com/2010/05/11/aunit-tutorials


AUnit requires a knowledge of Ada's OO features.

John

Re: Is Aunit helpful?

[Ada novice]

On Aug 8, 3:39 pm, John McCormick <mccorm...@cs.uni.edu> wrote:
>
> You may be interested in a video tutorial on AUnit produced by Daniel
> Bigelow (independent consultant,www.bigelow.ch).
>
> http://www.adacore.com/2010/05/11/aunit-tutorials
>
> AUnit requires a knowledge of Ada's OO features.
>
> John

Thanks for the link. I shall definitely take a look at the video. If
you are John McCormick, coauthor of the two books on Ada, well I would
like to thank you for writing these books on Ada, I have both of them
though I have only covered the first chapter in your Data Structures
book. OO features are described as from Chapter 2 and I'll study it in
due time.

Thanks
YC

Re: Is Aunit helpful?

[John McCormick]

On Aug 8, 12:10 pm, Ada novice <ycalleecha...@gmx.com> wrote:
>
> >http://www.adacore.com/2010/05/11/aunit-tutorials
>
> > AUnit requires a knowledge of Ada's OO features.
>
> Thanks for the link. I shall definitely take a look at the video. If
> you are John McCormick, coauthor of the two books on Ada, well I would
> like to thank you for writing these books on Ada, I have both of them
> though I have only covered the first chapter in your Data Structures
> book. OO features are described as from Chapter 2 and I'll study it in
> due time.
>
> Thanks
> YC

I'm very happy you have found my Ada textbooks useful!  It is always
nice when someone notices a book author's name.  Feel free to contact
me directly if you have any questions on the material in the books.
The OO material in the data structures book is elementary.  Check out
the Barnes or Ben-Ari books for an in depth treatment.  Should you
want to go on to learn about Ada's concurrent features, I have a new
book coming out at the end of the year: Building Parallel, Embedded,
and Real-Time Applications with Ada, www.cambridge.org/9780521197168
It is aimed at introducing the topics to novices with a background in
sequential programming.

Re: Is Aunit helpful?

[Michael R]

On Aug 9, 10:05 am, John McCormick <mccorm...@cs.uni.edu> wrote:
> On Aug 8, 12:10 pm, Ada novice <ycalleecha...@gmx.com> wrote:
>
>
>
> > >http://www.adacore.com/2010/05/11/aunit-tutorials
>
> > > AUnit requires a knowledge of Ada's OO features.
>
> > Thanks for the link. I shall definitely take a look at the video. If
> > you are John McCormick, coauthor of the two books on Ada, well I would
> > like to thank you for writing these books on Ada, I have both of them
> > though I have only covered the first chapter in your Data Structures
> > book. OO features are described as from Chapter 2 and I'll study it in
> > due time.
>
> > Thanks
> > YC
>
> I'm very happy you have found my Ada textbooks useful!  It is always
> nice when someone notices a book author's name.  Feel free to contact
> me directly if you have any questions on the material in the books.
> The OO material in the data structures book is elementary.  Check out
> the Barnes or Ben-Ari books for an in depth treatment.  Should you
> want to go on to learn about Ada's concurrent features, I have a new
> book coming out at the end of the year: Building Parallel, Embedded,
> and Real-Time Applications with Ada,www.cambridge.org/9780521197168
> It is aimed at introducing the topics to novices with a background in
> sequential programming.

Hi,

I think the link to this new book should be

http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521197168

Take care,
Michael.

Re: Is Aunit helpful?

[Ada novice]

On Aug 9, 7:23 pm, Michael R <mich...@zanyblue.com> wrote:
> I think the link to this new book should be
>
> http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521197168
>
> Take care,
> Michael.

Thanks for the link. As you pointed out, maybe Aunit is discussed
there. I couldn't find a detailed table of contents nor an index.
Maybe this is because the book is not yet out. Perhaps Prof. McCormick
would be kind to provide some more information on this forthcoming
book.


YC

Re: Is Aunit helpful?

[John McCormick]

On Aug 10, 3:33 am, Ada novice <ycalleecha...@gmx.com> wrote:
> On Aug 9, 7:23 pm, Michael R <mich...@zanyblue.com> wrote:
>
> > I think the link to this new book should be
>
> >http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521197168
>
> > Take care,
> > Michael.
>
> Thanks for the link. As you pointed out, maybe Aunit is discussed
> there. I couldn't find a detailed table of contents nor an index.
> Maybe this is because the book is not yet out. Perhaps Prof. McCormick
> would be kind to provide some more information on this forthcoming
> book.
>
> YC

Sorry, nothing on AUnit in this book.

Re: Is Aunit helpful?

[Mark Lorenzen]

On 9 Aug., 19:05, John McCormick <mccorm...@cs.uni.edu> wrote:
>
> I'm very happy you have found my Ada textbooks useful!  It is always
> nice when someone notices a book author's name.  Feel free to contact
> me directly if you have any questions on the material in the books.
> The OO material in the data structures book is elementary.  Check out
> the Barnes or Ben-Ari books for an in depth treatment.  Should you
> want to go on to learn about Ada's concurrent features, I have a new
> book coming out at the end of the year: Building Parallel, Embedded,
> and Real-Time Applications with Ada,www.cambridge.org/9780521197168
> It is aimed at introducing the topics to novices with a background in
> sequential programming.

I would like to suggest also taking a look at "Concurrent and Real-
Time Programming in Ada" by Alan Burns and Andy Wellings.

- Mark L

Re: Is Aunit helpful?

[John McCormick]

On Aug 10, 4:45 am, Mark Lorenzen <mark.loren...@gmail.com> wrote:
>
> I would like to suggest also taking a look at "Concurrent and Real-
> Time Programming in Ada" by Alan Burns and Andy Wellings.
>
> - Mark L

Excellent recommendation.  The Burns and Wellings book is the ultimate
reference and guide to these topics.  My copy is always within easy
reach and I keep the "prior edition", Concurrency in Ada, at home.
Like my forthcoming book, it is published by Cambridge University
Press.  Both the publisher and I felt that the B & W book is too
difficult for novices.  It was a real struggle for my Junior level
students with minimal Ada background to get through it.  We look at
Building Parallel, Embedded, and Real-Time Applications with Ada as a
prerequisite for the authoratative B & W book.

John

Re: Is Aunit helpful?

[Ada novice]

On Aug 9, 7:05 pm, John McCormick <mccorm...@cs.uni.edu> wrote:
> I'm very happy you have found my Ada textbooks useful!  It is always
> nice when someone notices a book author's name.  Feel free to contact
> me directly if you have any questions on the material in the books.
> The OO material in the data structures book is elementary.  Check out
> the Barnes or Ben-Ari books for an in depth treatment.  Should you
> want to go on to learn about Ada's concurrent features, I have a new
> book coming out at the end of the year: Building Parallel, Embedded,
> and Real-Time Applications with Ada,www.cambridge.org/9780521197168
> It is aimed at introducing the topics to novices with a background in
> sequential programming.

I do remember author names of books that have caught my attention :).
I have been looking at around 15 books (published since 1980) in Ada
and your two books are certainly very accessible to anyone wishing to
learn Ada well. I'm a PhD student in mechanical engineering wishing to
learn and use Ada for scientific numerical programming. So far, Ada
with its strong typing has enabled me to make much less mistakes than
I was doing earlier with C. And I have also learned a few aspects of
software engineering. As my interest is in scientific numerical
programming, only the first half or so of your Data Structures book is
relevant to me as I see it today. I wish that authors in future Ada
books to also focus a bit more on the numerics annex of Ada 05 and
give some useful examples which can help someone to build his/her
mathematical library.

I know Ben-Ari books. They are also well-written though at this
instant in time seem a bit advanced for me in some parts of the books.
Barnes' books are perhaps more challenging. What's make them
challenging in my opinion is the lack of complete examples but I
understand that complete examples would make less space for more
discussions. I'm not at this moment into concurrent features but I'm
certain that your forthcoming book will be pleasurable to read and I
would definitely read the book at some point in the future. Studying a
book on concurrency will in my opinion help to break one's habit of
always finding a solution using sequential logic that we tend to pick
up from other programming languages around and thus will surely help
to build better codes. I shall contact you directly if I have
questions relating to your books.

Thanks.
YC

Re: Is Aunit helpful?

[John McCormick]

On Aug 10, 12:32 pm, Ada novice <ycalleecha...@gmx.com> wrote:
> On Aug 9, 7:05 pm, John McCormick <mccorm...@cs.uni.edu> wrote:
>
> > I'm very happy you have found my Ada textbooks useful!  It is always
> > nice when someone notices a book author's name.  Feel free to contact
> > me directly if you have any questions on the material in the books.
> > The OO material in the data structures book is elementary.  Check out
> > the Barnes or Ben-Ari books for an in depth treatment.  Should you
> > want to go on to learn about Ada's concurrent features, I have a new
> > book coming out at the end of the year: Building Parallel, Embedded,
> > and Real-Time Applications with Ada,www.cambridge.org/9780521197168
> > It is aimed at introducing the topics to novices with a background in
> > sequential programming.
>
> I do remember author names of books that have caught my attention :).
> I have been looking at around 15 books (published since 1980) in Ada
> and your two books are certainly very accessible to anyone wishing to
> learn Ada well. I'm a PhD student in mechanical engineering wishing to
> learn and use Ada for scientific numerical programming. So far, Ada
> with its strong typing has enabled me to make much less mistakes than
> I was doing earlier with C. And I have also learned a few aspects of
> software engineering. As my interest is in scientific numerical
> programming, only the first half or so of your Data Structures book is
> relevant to me as I see it today. I wish that authors in future Ada
> books to also focus a bit more on the numerics annex of Ada 05 and
> give some useful examples which can help someone to build his/her
> mathematical library.
>
> I know Ben-Ari books. They are also well-written though at this
> instant in time seem a bit advanced for me in some parts of the books.
> Barnes' books are perhaps more challenging. What's make them
> challenging in my opinion is the lack of complete examples but I
> understand that complete examples would make less space for more
> discussions. I'm not at this moment into concurrent features but I'm
> certain that your forthcoming book will be pleasurable to read and I
> would definitely read the book at some point in the future. Studying a
> book on concurrency will in my opinion help to break one's habit of
> always finding a solution using sequential logic that we tend to pick
> up from other programming languages around and thus will surely help
> to build better codes. I shall contact you directly if I have
> questions relating to your books.
>
> Thanks.
> YC

Thanks for the kind words about my books.  It is those little things
that keep an author going.  The first edition of Programming and
Problem Solving with Ada did have a chapter devoted to numerics.  Or
what constituted numerics in Ada 83.  I had a great story about the
Patriot missle failure to motivate the need to understand what
Freshman see as a useless topic.  But when the publisher wanted to
shorten the text for the 2nd edition, they polled the schools using
the book and found that none covered the numberics chapter.

Great to see a non CS type interested in Ada.  My 1977 PhD is in
geology.  I got into computing through computational crystallography
and into embedded systems through the laboratory control of crystal
deformation experiments.  Started off with building analog control
computers and then made the leap to digital.  As the high pressure
apparatus beside which I sat contained the potential energy equavalent
to several sticks of dynamite I became very concerned with software
safety.

John

Re: Is Aunit helpful?

[Ada novice]

On Aug 12, 5:04 pm, John McCormick <mccorm...@cs.uni.edu> wrote:
>But when the publisher wanted to
> shorten the text for the 2nd edition, they polled the schools using
> the book and found that none covered the numberics chapter.

Thanks. Yes I read about this in the beginning of your textbook. Maybe
you could consider to post that particular numerics chapter on your
website. Of course this could depend on getting the publisher's
consent also; It could however be an easier task since this chapter is
from an earlier edition of your book. I believe that many students who
learn Ada would land on your website at one point or another and
finding some notes devoted to numerics would definitely encourage the
use of Ada in scientific programming.

There are quite a few good books on Ada out there and I try to read
those that contain software engineering aspects as well. Using
packages, stubs, etc make the final "main" code less cluttered this
has helped me a lot in writing "correct" programs. It's unfortunate
that non CS students typically have no (good) knowledge of software
engineering features. In 2005 I started learning LaTeX and thought
that no other computer-related stuff would ever have such positive
impact again on my life. Now I realize that I was wrong; Ada is
bringing much into my life and I'm already starting to learn SPARK.

Cheers
YC

Re: Is Aunit helpful?

[Yannick Duchêne (Hibou57)]

Le Wed, 04 Aug 2010 16:19:45 +0200, Ada novice <posts@gmx.us> a écrit:

> Hi, being a beginner in Ada and in software engineering, I came across
> a textbook mentioning aunit for writing test drivers. I write quite
> simple Ada programs and was wondering if it's worth to learn aunit and
> use it even on simple programs. I understand that it can be a time-
> saver.
I've never used AUnit and your thread was the opportunity to get time to  
learn about it.

Just came to this link which seems nice as a starting point:
http://libre.adacore.com/wp-content/files/auto_update/aunit-docs/aunit.html
This is not only about changes in a last version, this is also an  
accessible introduction.