Re: float with 24-bit resolution

[Mark Johnson <mark_h_johnson@raytheon.com>]

Ludovic Brenta wrote:
> 
> Mark Johnson <mark_h_johnson@raytheon.com> writes:
> > [snip explanation of integer based fixed point decimal values]
> 
> What you describe is a second-best solution, and by your own accord
> you only used it because the target hardware did not support
> packed-decimal representations.  Your divide-by-10 routine was the
> weak point; 10 happens not to be a power of two, and therefore any
> division of a binary number by 10 is bound to introduce rounding and
> imprecision.  
Perhaps you don't understand the solution I proposed. Using the example
I provided, to get the digits for display of 0.42, you would:
 - divide 42 by 10, getting result 4 and remainder 2 (last digit)
 - note that 4<10, getting result 0 and remainder 4 (second digit)
 - generate the decimal point
 - generate the zero
and display the value 0.42. No round off errors or any imprecision. This
approach can be generalized to any fixed point decimal type. Depending
on the range of values / precision required, it may require the
occasional use of double precision values and integer division that
produces both the result and remainder.

As I mentioned before, you certainly have to do the "right thing" when
it comes to doing arithmetic and conversions to other forms, but fixed
point decimal arithmetic without packed decimal data types can be done
without introducing any errors.

> You may minimise the impact of that by doing the
> division as late as possible, e.g. for display purposes; this is
> probably what made the solution acceptable to you.  However, if your
> program must write to a text file that serves as input to another
> program (and then another, etc.), then the loss of precision quickly
> becomes unacceptable.
Once you understand how to generate the correct values w/o any error,
you will find that this statement does not apply.
  --Mark