Re: Languages don't matter. A mathematical refutation

[brbarkstrom@gmail.com]

I find myself disagreeing with Dr. Kazakov and agreeing more with
Dr. Martinez.  In the following scenario, I've tried to be careful
about separating the statistical issues from the information that
is needed to distinguish why two samples differ.  From that standpoint, 
a statistician can distinguish whether two sampled distributions are likely to come from the same development methodology without knowing anything about the languages, team quality, organizational structure, and the related items that
have consumed the discussion in much of this thread.

Consider a consortium of development organizations.  Some of them use methodology "A".  Another group uses methodology "C".  All members of
the consortium keep records of their total maintenance costs for ten
years.  The consortium members agree it might be useful to share these two pieces of information.  Accordingly, they build a list containing just three pieces of information for each member: name of the organization, the development methodology, and the ten year cost.
For example, the first four items in this list might contain the elements
	Company 1,	"A",		$10,000
	Company 2,	"A",		$25,000
	Company 3,	"C",		$50,000
	Company 4,	"C", 	        $5,000
and so on.

Company Q's CEO hires a statistician to sample this list.  The statistician doesn't want to read the whole list, so he arranges to sample it so that it creates two new lists.  The first of these contains the information only from companies that use method "A".  The second contains information only from 
companies that use method "C".

To assure that the samples selected from the original list are not
duplicated, the statistician selects the samples by starting with a random initial index.  He selects the line with that starting index.  It contains just tho three properties in the sample above.  If the methodology is "A", the information goes into the new list with just "A" information.  If the methodology is "C",it goes into another new list with just the "C" information.

The statistician's algorithm then lets an integer pseudorandom number generator select a new number and adds that to the index for selecting the next sample from the original list.  The algorithm treats the three fields from the new sample in the same way as the first sample.

This process continues until the statistician is satisfied he has enough samples.  [This will probably be somewhere in the vicinity of 20, although it might be more.]

The statistician now has two lists - selected at random from the original list.  At this point, he can apply standard statistical tests to the numerical values for cost to see if the distributions differ significantly.  If he wishes to treat the numerical values as integers, the standard test is the Chi-Square Test.  Because the number of samples with methodology "A" is likely to differ from the number using "C", the standard test is a two-sided Chi-Square.  If he wants to treat the numerical values as floating point (or real), then the standard test is the Kolmogorov-Smirnov test.  Again, there is a one-sided version (for testing whether the distribution is different from a known distribution) and a two-sided version.

Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T., 1986: Numerical Recipes: The Art of Scientific Computing, Cambridge University Press, Cambridge, UK, pp. 469-475

provides a very readable introduction to all four of these tests.
Note that their copyright is still enforced, so be careful about simply lifting the code from this source (or later editions).

Knuth, D. E., 1998: The Art of Computer Programming, Vol. 2/Seminumerical Algorithms, Third Edition, Addison-Wesley, Boston, MA, pp. 41-58

provides the full derivation of the K-S distribution, as well as other tests for whether a random number generator is producing reliably random output.  The K-S test is covered on pp. 48-55 (including the algorithm in full detail).

At the end of the testing, the statistician can report to the CEO on whether the distribution of ten-year costs for methodology "A" has a statistically significant difference from the distribution of costs with methodology "B".

Note that the conclusion says nothing about why the two methodologies differ.  If they are different, maybe it's because the companies using "A" are using a higher level language.  Maybe it's because the companies using "A" are more experienced or have "higher quality" developers.  Those attributes are irrelevant to the question the CEO asked of the statistician - which was "are these two methodologies drawn from significantly different distributions?".

In order to understand the causes of the differences, the statistician would need to undertake a much broader investigation with many more samples and request many more pieces of information from the members of the consortium.  Probably the most widely-known example of this kind of exploration is found in

Boehm, B. W., Abts, C., Brown, A. W., Chulant, S., Clark, B. K., Horowitz, E., Madachy, R., Reifer, D. and Steece, B., 2000: Software Cost Estimation with COCOMO II, Prentice Hall PTR, Upper Saddle River, NJ

Boehm has been conducting research in understanding why software development with different methodologies creates differences in the cost of the software that's delivered for about 30 years.  The cover of this book shows twenty one factors that he's parameterized as having an effect on cost - and the contents provide all the details needed to estimate the costs.  Unfortunately for the discussion we've been having, Boehm puts language selection into a parameter he calls LTEX, which is "a measure of the level of programming language and software experience of the project team developing the software system or subsystem." [Boehm, et al., pp. 48-49]

It seems clear that it is quite possible to make a random selection
of development approaches even though the methodology of each team is fixed and determinate at the time of the sampling.  The math of demonstrating the statistical significance of difference between two sample distributions is also quite clear and well-established.

Bruce B