Re: OT: Incremental statistics functions

[tmoran@acm.org]

>Yes, be very, very careful.  The problem/issue is that for the general
>case computing the mean and then the standard deviation has very good
>mathematical properties.  Computing them incrementally does not.  If you
>have an 'expected' value for the mean (call it u) you can use it to
>accumulate (Xi-u)**2 and if u is 'close' to the mean, the numerical
  If your incoming data is random, you could use the average of the first
bunch of values as u.  Or if N is large and the incoming data is really
random, you might want to act like a pollster and just take a subsample.
If the data has a pattern, it might be worth using other Monte Carlo
techniques.
  Even the average can be a problem if N is large.  Say you use 6 decimal
digit floating point for the running sum and the first million values are
all 10.0, followed by a million 9.0s so the correct average is 9.5 After
adding the million 10.0s your floating sum will lose its units position
and all the additions of 9.0s will change nothing.  So after all the data
is in, the sum is ten million, divided by N of 2 million, gives you a
value of 5.0, not the correct 9.5
  If you are just calculating the average and variance (standard deviation
squared) of the heights of people in your office, all these considerations
are irrelevant and you probably don't need a statistician. :)