Re: Is Aunit helpful?

[Midoan <midoan.ses@gmail.com>]

On Aug 15, 1:10 pm, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> Midoan <midoan....@gmail.com> writes:
> > On Aug 14, 6:57 am, Stephen Leake <stephen_le...@stephe-leake.org>
> > wrote:
> >> "Yannick Duchêne (Hibou57)" <yannick_duch...@yahoo.fr> writes:
>
> >> > About AUnit: just seen about what it is, how it is set up and how it
> >> > works. Seems a question is still pending : “how to be sure the test
> >> > cover  all relevant case ?”. I do not see a way to be sure testing
> >> > cover all  cases.
>
> >> Correct, AUnit does not do that. gcov does, although I have not used it
> >> very much. It can be difficult to use the output of gcov.
>
> >> > That is the main limitation of this kind of approach.
>
> >> What alternative approaches provide coverage information?
>
> > FYI, note that with Mika (http://www.midoan.com/), the automatic test
> > data  generator for Ada, it is possible to take in your existing test
> > cases, check the coverage achieved, and automatically generate missing
> > test inputs and expected test result
>
> How can a tool possibly generate expected results? If it reads the code,
> it can only generate the results that the code _will_ produce. But
> that's the opposite of a test; the expected results are what the code
> _should_ produce, based on some other spec (not the Ada spec). A testing
> process _must_ assume the code is wrong.

Yes of course you are right; and that's what we meant. What we meant
was that the code's result must always be validated externally of the
tool. Mika will simply ask the question "given these inputs, are those
expected outputs?". Those outputs, that Mika generates, will be
according to the code under test: it will include all side effects
(i.e. including non local variables changes) ... In our experience,
Mika will flag many side effects not necessarily intended by the
developers, never mind the designers ... If the code's outputs is
however confirmed by the human oracle then it can join a set test
cases suitable for regression testing. If the code is subsequently
changed then Midoan can flag changed behavior and/or complete the test
inputs set; thus being much useful during developement, maintenance
(i.e. regression) testing.

> If the spec is machine readable, then the tool has a chance. But I see
> no mention of machine readable specs on the midoan site.
>
> Hmm. If the Ada spec includes pre/postconditions (Ada 2012), then some
> meaningful tests can be generated, but the compiler will already do
> that.
>
> > (which can be validated to form new test cases automatically).
>
> validated by what? If that means "reviewed by a human", that might be
> ok. But there would be a very strong temptation to say "the tool must be
> right".
>
> > This can be done to achieve branch, decision or MC/DC coverage as
> > desired.
>

We are very open to inquiries about machine readable, commercially
sustainable (preferably!), specification languages as used in
combination with Ada (any specification language based on Ada's syntax
is particularly welcome :-) ). In fact disproving specified behavior
would be a very attractive proposition for us ... But, in our
experience, most specifications (and that includes code generated (!)
specifications used by fashionable model based verification techniques
as provided by Polyspace and such like) only capture a fraction of the
code's behavior; e.g. not all packages' variables are modeled. In
practice, and for example, Ada 2012's post-conditions are unlikely to
be used to capture the full effect of commercial, safety critical,
code ... That, to our eyes, is a major weakness in specification based
verification : the specification may say 'A' but the code may do 'A
and B' ... and trying to model 'B' in the specification typically
leads to  specifications as complex as the code!

So we, as a gist conclusion, at Midoan, still believe in the primacy
of human checked, tool assisted, systematic software testing.

> Generating a scaffold that gives coverage could be useful, but it must
> be completed manually, to ensure correct results.
>

Agreed, indeed.

> --
> -- Stephe

Regards,
The Midoan Team at http://www.midoan.com/