Re: OT: Incremental statistics functions

["Robert I. Eachus" <rieachus@comcast.net>]

Mats Karlssohn wrote:

> Basically I don't want to keep a (limited) buffer of samples but would like
> to add the values one at a time when they are calculated.
> 
> Probably I googled bad, since I didn't find anything I could understand.
> 
> Any suggestions please?

Yes, be very, very careful.  The problem/issue is that for the general 
case computing the mean and then the standard deviation has very good 
mathematical properties.  Computing them incrementally does not.  If you 
have an 'expected' value for the mean (call it u) you can use it to 
accumulate (Xi-u)**2 and if u is 'close' to the mean, the numerical 
characteristics will be fine.  (To put that technically, if your 
estimate u, is less than half a standard deviation from the sample mean 
you should have nothing to worry about.)

To put all this in perspective, say you are monitoring daily low 
temperatures in New Hampshire in January.  There is no problem, the day 
to day differences are larger than difference between zero and the 
average.  Try the same thing in Iraq in July, and you won't do as well. 
  Even if you are monitoring low temperatures, eventually the difference 
between the sum of the squares and n times x-bar squared will be much 
smaller than those two numbers. And if you are using floating point, 
that ratio determines how much significance you have lost.  Your 
estimate of the variance or standard deviation may have just a few 
significant bits (one significant digit) or worse, no significant bits 
or digits.

You can guard against this to some extent by using IEEE double or 
extended for computing the sum of the squares, but that only postpones 
when you run out of significance, it doesn't prevent it.


-- 
                                           Robert I. Eachus

"The war on terror is a different kind of war, waged capture by capture, 
cell by cell, and victory by victory. Our security is assured by our 
perseverance and by our sure belief in the success of liberty." -- 
George W. Bush