Designing test suites (was: if statements)

[Kilgallen@eisner.decus.org.nospam (Larry Kilgallen)]

In article <mjxO5.9218$pq3.738093@news.flash.net>, "Ken Garlington" <Ken.Garlington@computer.org> writes:
> "Robert Dewar" <robert_dewar@my-deja.com> wrote in message
> news:8udasq$kn$1@nnrp1.deja.com...
> 
> : I suppose one could at least introduce SOME notion of
> : aesthetics, e.g. evaluate layout against precanned style
> : rules, or evaluate complexity using established metrics.
> 
> Given the presumed use of Ada to support highly reliable systems, I'd
> especially like to see a robot of this type attempt to "break" the solution
> by feeding it invalid inputs, etc. It might not be possible for something
> this simple, but once you get up to more complete solutions, I think bonus
> points should be awarded to solutions that implement error recovery, etc.

One approach that has struck me lately (in my real work) is striving
to get a test suite that achieves 100% code path coverage on some
_different_ implementation, and then running that test suite against
the implementation under test.

For the student problem case (as you say, something not so simple),
it might be possible to refine the test each year to achieve 100%
code path coverage on every solution that has ever been submitted
(perhaps even the incorrect ones).