Re: Floating point accuracy

["Martin Dowie" <martin.dowie@gecm.com>]

doh!
It's my eyes that don't have the necessary significant digits!

As Markku points out, the values for "ls" _are_ different - just
not where I had expected! In fact they are exactly 1000ns different,
as I should have expected (the last 3 digits are the same, and
confusing me!)

But, Markku, the 'Floor is doing exactly what I want it to do... :-)


"M. Kotiaho" <kotiaho@m_a_i_l.com> wrote in message
news:3A8BF422.69723366@m_a_i_l.com...
> Martin Dowie wrote:
>
> Yes ... look more closely at your results ... the values you print for
"ns"
> differ in one digit (last non-zero digit).  The values for "ls" are also
> different
> (the last value has ...66..., the previous one has ...65...).  It is only
for
> "ms"
> and "MS" that the values are the same.
>
> You should not expect otherwise, since you are losing information when
> you take the 'floor operation in computing Float_MS.  It has nothing to do
> with the number of significant digits (since you have enough).
>
> Here is the math:
>
> 'Last - 1:
> Float_MS = Floor( (2**32 - 2)*1000 / 2**32 ) = Floor( 1000 - 2000/2**32 )
= 999
>
> 'Last:
> Float_MS = Floor( (2**32 - 1)*1000 / 2**32 ) = Floor( 1000 - 1000/2**32 )
= 999
>
> ...<snip>...
>
> >
> > test 'Last - 1
> > ns =  4.29496729400000E+12
> > ms =  9.99000000000000E+02
> > MS =  4.29067232870400E+12
> > ls =  4.29496529600000E+09
> >
> > ms => 999 ls => 4294965296
> > test 'Last
> > ns =  4.29496729500000E+12
> > ms =  9.99000000000000E+02
> > MS =  4.29067232870400E+12
> > ls =  4.29496629600000E+09
> >
> > ms => 999 ls => 4294966296
>
> Regards,
> Markku Kotiaho
>
>
>