Consulting

Results 1 to 4 of 4

Thread: Inaccurate results from formula

  1. #1
    Moderator VBAX Wizard Aussiebear's Avatar
    Joined
    Dec 2005
    Location
    Queensland
    Posts
    5,059
    Location

    Inaccurate results from formula

    I've been amending gps marks from Time Degrees to Decimal degrees and the results are not quite what I was expecting. The formula I use is "=Left(A1,3) + Mid(A3,4,2)/60 + Right(A1,3)/60/60" which strips strings based on Degrees.Minutes.Seconds

    A string such as 150.50.000 is returning 150.00833333 which is clearly wrong if anything it should be 150.83333333 since the Left function returns 150, the Mid function is meant to return 0.83333333, and the Right function returns zero. What could be happening here?
    Remember To Do the Following....
    Use [Code].... [/Code] tags when posting code to the thread.
    Mark your thread as Solved if satisfied by using the Thread Tools options.
    If posting the same issue to another forum please show the link

  2. #2
    Knowledge Base Approver VBAX Wizard
    Joined
    Apr 2012
    Posts
    5,645
    I'd say, use

    Mid(A3,5,2)/60

  3. #3
    Moderator VBAX Wizard Aussiebear's Avatar
    Joined
    Dec 2005
    Location
    Queensland
    Posts
    5,059
    Location
    Never mind. Too many hours in the hot sun I guess. Formula is incorrect and should read "=Left(A1,3)+Mid(A1,5,2)/60+Right(A1,3)/60/60"
    Remember To Do the Following....
    Use [Code].... [/Code] tags when posting code to the thread.
    Mark your thread as Solved if satisfied by using the Thread Tools options.
    If posting the same issue to another forum please show the link

  4. #4
    Moderator VBAX Wizard Aussiebear's Avatar
    Joined
    Dec 2005
    Location
    Queensland
    Posts
    5,059
    Location
    Thanks snb
    Remember To Do the Following....
    Use [Code].... [/Code] tags when posting code to the thread.
    Mark your thread as Solved if satisfied by using the Thread Tools options.
    If posting the same issue to another forum please show the link

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •