Page 1 of 1

Float Rounding Problem

Posted: Fri Apr 22, 2022 2:04 pm
by wayne_millard
Hi Matrix Team,

Im having some problems with Float Rounding. I have attached an example file.
What am i doing wrong? Only one of the roundings seem to do something.

Thanks,

Wayne M
Float Rounding.fcsx
Float Rounded Problem
(8.34 KiB) Downloaded 120 times

Re: Float Rounding Problem

Posted: Fri Apr 22, 2022 2:42 pm
by BenR
Hello Wayne,

It looks like it's working in the simulation, but maybe the precision of storing and displaying the number is not ideal. I beleive this is the limitation of the float format rather than anything else.
Rounding.jpg
Rounding.jpg (18.33 KiB) Viewed 1918 times
This is a nice tool as it allows you to enter a real number and it will show you the floating point representation of the number.

https://www.h-schmidt.net/FloatConverter/IEEE754.html

100.7 is actually stored as 100.6999969482421875
100.68 as 100.68000030517578125
etc

Re: Float Rounding Problem

Posted: Fri Apr 22, 2022 6:14 pm
by wayne_millard
Hi everyone

What I wanted was to only display to two places after the decimal point.

I thought in the past when you did a round to two decimal places you would loose all the other numbers.

So how do you do it.

Thanks
Wayne m

Re: Float Rounding Problem

Posted: Fri Apr 22, 2022 6:27 pm
by mnfisher
Hi Wayne,

Try using str = FloatToStr$(val, 2)
Then display the string str.

Martin

Re: Float Rounding Problem

Posted: Sun Apr 24, 2022 1:35 pm
by wayne_millard
Martin, Thanks I forgot about that.

Wayne