I read inputs to a VBA macro from a worksheet, calculate values with double precision variables. I display the result back on the worksheet - and they have only 7 significant digits instead of 14.
Question: What operations can cause a double precision variable to have only 7 significant digits?
Make sure all of your intermediate variables and constants are double precision
Can you post example?
---------------------------------------------------------------------------------------------------------------------
Paul
Remember: Tell us WHAT you want to do, not HOW you think you want to do it
1. Use [CODE] ....[/CODE ] Tags for readability
[CODE]PasteYourCodeHere[/CODE ] -- (or paste your code, select it, click [#] button)
2. Upload an example
Go Advanced / Attachments - Manage Attachments / Add Files / Select Files / Select the file(s) / Upload Files / Done
3. Mark the thread as [Solved] when you have an answer
Thread Tools (on the top right corner, above the first message)
4. Read the Forum FAQ, especially the part about cross-posting in other forums
http://www.vbaexpress.com/forum/faq...._new_faq_item3
you can convert the numbers as Currency or Decimal while performing calculation in VBA.
these two are more accurate than Double datatype.
Thanks, I will test those.Originally Posted by arnelgp
I don't understand the currency data type. I tested it and it seems to always round decimal fractions to 4 places. Perhaps there is a technique to get around that for general usage?
Sorry but I would have to disagree there when using my Macs. We had major problems converting Degrees/Minutes to Eastings/Northings for Vincenty formula until all values were converted using Double datatypes
Remember To Do the Following....
Use [Code].... [/Code] tags when posting code to the thread.
Mark your thread as Solved if satisfied by using the Thread Tools options.
If posting the same issue to another forum please show the link
I am tracing values through my VBA code which I suspend with "stop". Is it correct determine significant digits by counting the digits in the popup that occurs when I hold the cursor over a variable? Is there a better way?
Sub M_snb() MsgBox 1 / 3 & vbLf & Len(CStr(1 / 3)) End Sub
Posting the code that causes the truncation
Are you sure that you're looking at the Display of the number and not the full precision?
Capture.JPG
Last edited by Paul_Hossler; 06-11-2022 at 04:47 PM.
---------------------------------------------------------------------------------------------------------------------
Paul
Remember: Tell us WHAT you want to do, not HOW you think you want to do it
1. Use [CODE] ....[/CODE ] Tags for readability
[CODE]PasteYourCodeHere[/CODE ] -- (or paste your code, select it, click [#] button)
2. Upload an example
Go Advanced / Attachments - Manage Attachments / Add Files / Select Files / Select the file(s) / Upload Files / Done
3. Mark the thread as [Solved] when you have an answer
Thread Tools (on the top right corner, above the first message)
4. Read the Forum FAQ, especially the part about cross-posting in other forums
http://www.vbaexpress.com/forum/faq...._new_faq_item3
Is the column wide enough?
p45cal
Everyone: If I've helped and you can't be bothered to acknowledge it, I can't be bothered to look at further posts from you.
If the truncated cell is the output of a formula, you can display with full (all that is available) precision by selecting the cell (e.g. C1) click in the formula bar, and pressing F9
Capture.JPG
Capture2.JPG
---------------------------------------------------------------------------------------------------------------------
Paul
Remember: Tell us WHAT you want to do, not HOW you think you want to do it
1. Use [CODE] ....[/CODE ] Tags for readability
[CODE]PasteYourCodeHere[/CODE ] -- (or paste your code, select it, click [#] button)
2. Upload an example
Go Advanced / Attachments - Manage Attachments / Add Files / Select Files / Select the file(s) / Upload Files / Done
3. Mark the thread as [Solved] when you have an answer
Thread Tools (on the top right corner, above the first message)
4. Read the Forum FAQ, especially the part about cross-posting in other forums
http://www.vbaexpress.com/forum/faq...._new_faq_item3
Thanks. That saved the work of widening the columns and using the arrows on the [Home] menu to expand the number until you start getting all zeros - and then guessing which zeros are significant.
By the way, can anyone explain why those arrows point in the wrong direction?
The arrows point in the direction that you wish to move the decimal point.
Remember To Do the Following....
Use [Code].... [/Code] tags when posting code to the thread.
Mark your thread as Solved if satisfied by using the Thread Tools options.
If posting the same issue to another forum please show the link
Thanks
OK, I see where an abstract programmer would think of where the decimal point goes. I think more visually, and see adding the digits at the end of a decimal fraction as expanding to the right. I often press the right arrow wrongly.
I never thought of it that way -- I always thought, "Damn I wanted the other one!" I learned something today
---------------------------------------------------------------------------------------------------------------------
Paul
Remember: Tell us WHAT you want to do, not HOW you think you want to do it
1. Use [CODE] ....[/CODE ] Tags for readability
[CODE]PasteYourCodeHere[/CODE ] -- (or paste your code, select it, click [#] button)
2. Upload an example
Go Advanced / Attachments - Manage Attachments / Add Files / Select Files / Select the file(s) / Upload Files / Done
3. Mark the thread as [Solved] when you have an answer
Thread Tools (on the top right corner, above the first message)
4. Read the Forum FAQ, especially the part about cross-posting in other forums
http://www.vbaexpress.com/forum/faq...._new_faq_item3
OMG... is this an Aussie thing?
Remember To Do the Following....
Use [Code].... [/Code] tags when posting code to the thread.
Mark your thread as Solved if satisfied by using the Thread Tools options.
If posting the same issue to another forum please show the link
Math operations with Currency Types are limited to 4 decimal places to prevent rounding issues with "Smaller" Decimal values.
4 decimal places are sufficient for mathematics for all currencies that use 1/100 units. Ie: $1.12
I expect the student to do their homework and find all the errrors I leeve in.
Please take the time to read the Forum FAQ
Posting Permissions
Reply With Quote
