PDA

View Full Version : Wierd Calculation Issue - VBA- Excel (2007)



agraano
07-31-2010, 11:37 AM
MsgBox (821.557 - 456.554) - 365.003
Cells(9, 1).Value = 800.557 - 400.111 - 400.446
Cells(10, 1).Value = 800.225 - 400.111 - 400.114

The above is some test codes (i created to debug my original code).

Technically, the result of the calculation should be a zero. however, i get something like the following:

0.00E+00
5.6843418860808E-14

Can anyone help to get the calculation right?

mikerickson
07-31-2010, 04:20 PM
No. Roundoff error like this is inherent in all computers. It has to do with how a computer converts decimal to and from binary.

agraano
08-01-2010, 02:18 AM
@mikerickson: Is there a workaround to this problem?

In the code that I am trying to write, its important that the calculations such as this is accurate upto 4 decimal point...

mikerickson
08-01-2010, 01:00 PM
You could use Application.WorksheetFunction.Round to force everything into 9 decimal places.
(9 because the product of two 4 dp values might string out to 8 dp.)