Page 1 of 1
Float Rounding Problem
Posted: Fri Apr 22, 2022 2:04 pm
by wayne_millard
Hi Matrix Team,
Im having some problems with Float Rounding. I have attached an example file.
What am i doing wrong? Only one of the roundings seem to do something.
Thanks,
Wayne M
Re: Float Rounding Problem
Posted: Fri Apr 22, 2022 2:42 pm
by BenR
Hello Wayne,
It looks like it's working in the simulation, but maybe the precision of storing and displaying the number is not ideal. I beleive this is the limitation of the float format rather than anything else.
- Rounding.jpg (18.33 KiB) Viewed 1918 times
This is a nice tool as it allows you to enter a real number and it will show you the floating point representation of the number.
https://www.h-schmidt.net/FloatConverter/IEEE754.html
100.7 is actually stored as 100.6999969482421875
100.68 as 100.68000030517578125
etc
Re: Float Rounding Problem
Posted: Fri Apr 22, 2022 6:14 pm
by wayne_millard
Hi everyone
What I wanted was to only display to two places after the decimal point.
I thought in the past when you did a round to two decimal places you would loose all the other numbers.
So how do you do it.
Thanks
Wayne m
Re: Float Rounding Problem
Posted: Fri Apr 22, 2022 6:27 pm
by mnfisher
Hi Wayne,
Try using str = FloatToStr$(val, 2)
Then display the string str.
Martin
Re: Float Rounding Problem
Posted: Sun Apr 24, 2022 1:35 pm
by wayne_millard
Martin, Thanks I forgot about that.
Wayne