Re: Is strong typing worth the cost?

["Marin David Condic" <dont.bother.mcondic.auntie.spam@[acm.org>]

Experiments in software engineering could get very expensive. Maybe not as
expensive as high-energy particle physics, but pretty expensive. In order to
be meaningful, they would also need to be of sufficient size that they would
become very time consuming - making possible results rather dated.

Consider that to test the hypothesis: "Strong Typing In A Language Improves
Productivity And Reduces Errors" you would need the following:

1) A set of requirements for a project that should result in software of
100,000 slocs or better. (Small projects aren't likely to show much benefit,
nor are they realistically representative of real world programming.)

2) Two designs to those requirements that are as close to identical as
possible. One based on a strong typing model and the other relying on raw
machine types. (This may not even be possible since that really reflects two
different design philosophies.)

3) Two language compilers that support identical languages except that in
one, you have strong type checking and in the other, you have just machine
types with automatic conversions, etc. (You can't just say "use the standard
types in Ada" because even those are strongly typed and eliminate a whole
class of errors on their own. You'd need to have something like Ada with C
type checking. I'm not convinced you could get there without destroying
other aspects of the language that would then alter the validity of the
results.)

4) Two teams of reasonably similarly experienced engineers (Does the test
prove the hypothesis for inexperienced students or for experienced
engineers? Did they know the languages in question before they started the
project or did they have to learn them? It isn't simple, is it?) The
engineers would have to be available full-time for the duration of the
project (6 months, to be realistic?) so start thinking of salaries and how
to get engineers to forgo/interrupt promising careers to be part of the
study.

5) Identical development environments - the only variable being the language
support for the type model. (Is that fair? What about compiler bugs? Could
they account for delays, reported errors, etc?)

6) A well defined development process that is adhered to by both groups.
(Errors are injected by sloppy CM, lack of code reviews, etc. How tightly is
strong typing tied to development process anyway? Are weakly typed languages
used in places with no identifiable process and strongly typed languages
used by developers with well organized and followed procedures?)

7) A rigorous test suite that can be applied to both end products to
determine compliance with requirements and existence of errors. (You can't
write a test suite to detect all errors, but its got to be thorough enough
to demonstrate the most eggregious ones or its not going to demonstrate the
hypothesis.)

Even when you get done with all that, we can still cast serious stones at
whatever results are demonstrated:

1) Its only one trial - To reflect reality, it ought to be conducted lots of
times with different teams at different skill levels.

2) Its only one problem domain. Type checking might be real important in one
problem domain and not a big deal in another.

3) Whatever the project looks like, its artificial and doesn't reflect the
kinds of things that need to be done or dealt with in a real project.
(Things like shifting requirements, vague problem definitions, needing to
meet deadlines, etc.)

4) Whatever the results are, they reflect the technology available at the
start of the research project (6 months? A year before the results are
ready?) and that is a long time in Computer Years. Would better debuggers,
faster computers, automatic code generators, analysis tools, etc. that are
available today have changed the outcome?

All in all, its a lot simpler and less costly to deal with questions like
"What happens if I drop different sized cannonballs off a tall building?"
than it is to ask questions about software engineering. Not that I'd be
against asking those questions and trying to come up with experiments to
investigate them, but you've got to admit that doing so in anything like a
scientific manner would cost a LOT of money. Its probably worth doing, but
who would pay? (I'd personally love to have a research grant to conduct a
study like that. It would get me out of hacking C code for a living! :-)

Question: What I outlined here is pretty rigorus. Would you accept as being
useful or valid some kind of experiment that didn't require such controlled
conditions?

MDC
--
Marin David Condic
Senior Software Engineer
Pace Micro Technology Americas    www.pacemicro.com
Enabling the digital revolution
e-Mail:    marin.condic@pacemicro.com


"dmjones" <derek@NOSPAMknosof.co.uk> wrote in message
news:Xns9200B524E1781derekknosofcouk@62.253.162.109...
>
> Is experimental evaluation a dead duck in software engineering?
>
> http://citeseer.nj.nec.com/lukowicz94experimental.html