 Author Topic: is there a way to stop rounding  (Read 155 times)

small no more

• Sr. Member
•    • Posts: 251
• Ask and you will receive, Seek and you will find
•  is there a way to stop rounding
« on: February 20, 2013, 09:09:33 PM »
hello

is it possible to stop rounding of a number?

value = 12345678901234.6789
print value

prints    1234567891234.68 instead of the actual value 1234567891234.6789
A good teacher must be able to put himself in the place of those who find learning hard.

Eliphas Levi

mcalkins

• Hero Member
•     • Posts: 1269
• •  Re: is there a way to stop rounding
« Reply #1 on: February 20, 2013, 10:40:20 PM »
Rounding is part of the nature of floating point. Also, remember that the numeric data types are binary, not decimal.

If you want to use floating point numbers, and avoid rounding, you must: Use a data type with sufficient precision. Avoid fractions whose denominators are not powers of 2.

For details of data type precision, see:
http://en.wikipedia.org/wiki/IEEE_754-1985
http://en.wikipedia.org/wiki/Extended_precision#x86_Extended_Precision_Format

In this case, 6789 / 10000:
6789 has prime factors 3, 31 and 73.
10000 has prime factors 2 and 5.
The fraction cannot be reduced so that the denominator would be a power of 2. Therefore 6789 / 10000 cannot be stored accurately in a binary floating point number of any precision.

Why does 1 / 3 repeat in decimal? Because the denominator has a prime factor (3) other than 2 or 5. To terminate in base 10, the reduced fraction must not have a denominator other than 2 or 5.

750 / 1000 terminates in both binary and decimal.
The reduced fraction is 3 / 4.
4 has a prime factor of 2. It does not have a prime factor other than 2, so it terminates in binary. It does not have a prime factor other than 2 or 5, so it terminates in decimal.

What can you do? Use an integer. 123456789012346789 fits in an _INTEGER64. You will not have any rounding with integers, but you can still have overflow problems. You could implement your own fixed point binary or decimal math operations, or use the existing operations, compensating where necessary. If you do a search, you should find related discussion on this subforum related to Visual BASIC's CURRENCY type.

http://www.qb64.net/forum/index.php?topic=5092.0
http://www.qb64.net/forum/index.php?topic=1551.0

Examples of favoring integers over fractions would include using integer pennies instead of fractional dollars, and using integer milliseconds instead of fractional seconds.

Regards,
Michael
« Last Edit: February 20, 2013, 11:42:05 PM by mcalkins »
The QBASIC Forum Community: http://www.network54.com/index/10167 Includes off-topic subforums.
QB64 Off-topic subforum: http://qb64offtopic.freeforums.org/

mcalkins

• Hero Member
•     • Posts: 1269
• •  Re: is there a way to stop rounding
« Reply #2 on: February 20, 2013, 11:09:22 PM »
Note also, that the STR\$ operation on floating point numbers involves its own rounding. (PRINTing a number implicitly involves an STR\$ operation.)

This demonstrates it:

Code: [Select]
DIM value AS _FLOAT
DIM i AS _UNSIGNED _INTEGER64
value = 12345678901234.9375##
PRINT STR\$(value), value
PRINT
PRINT value * 1024
i = value * 1024
PRINT i
value = value * 1024
i = value
PRINT i
PRINT
value = 12345678901234.94##
i = value * 1024
PRINT i

Note that the displayed 12345678901234.94 is rounded. However, the variable still contains the correct number. When we multiply by 1024, we get the exact number of 12641975194864576.

Note that 9375 / 10000 can be reduced to 15 / 16.
16 has a prime factor of 2, so, the fraction terminates in both binary and decimal.

Regards,
Michael
The QBASIC Forum Community: http://www.network54.com/index/10167 Includes off-topic subforums.
QB64 Off-topic subforum: http://qb64offtopic.freeforums.org/

codeguy

• Hero Member
•     • Posts: 3552
• what the h3ll did i name that code?
• •  Re: is there a way to stop rounding
« Reply #3 on: February 21, 2013, 07:58:55 AM »
or you can handle strings as numbers, either using a BIGINT library or roll your own.
http://denteddisk.forums-free.com/make-an-appointment-with-the-resident-code-guru-f34.html

Cyperium

• Hero Member
•     • Posts: 3285
• Knowledge is good, but understanding is better
• •  Re: is there a way to stop rounding
« Reply #4 on: February 22, 2013, 05:24:06 PM »
Quote from: small no more on February 20, 2013, 09:09:33 PM
hello

is it possible to stop rounding of a number?

value = 12345678901234.6789
print value

prints    1234567891234.68 instead of the actual value 1234567891234.6789

Brute force string manipulation method:
Code: [Select]
DIM value AS DOUBLE

'keep is the number of decimals you want to keep
keep = 2
value = 1000000.5444444449

decp = INSTR(STR\$(value), ".")
keepdec = LEN(STR\$(value)) - decp - 1
DO
IF decp > 0 AND VAL(MID\$(STR\$(value), decp + keepdec + 1, 1)) >= 5 THEN inc = 1
IF decp THEN value = VAL(MID\$(STR\$(value), 1, decp + keepdec))

IF inc THEN value = value + 1 / (10 ^ keepdec)

IF keepdec <= keep THEN EXIT DO
keepdec = keepdec - 1
LOOP

PRINT value
« Last Edit: February 22, 2013, 05:32:40 PM by Cyperium »
Venture - New Prototype, QB64 Editor v1.95b (linux compatible, if you compile the source).