precision issues with floating-point

  • Resolved ramanandmehta

    (@ramanandmehta)


    1 year, 2 months ago

    0.0166666667 * 60 = 1
    but the form is giving result 1.000000002
    why this extra zeros?
    I asked to AI and he says: It seems like you’re encountering precision issues with floating-point arithmetic in JavaScript. The value 0.0166666667 is not exactly representable in the binary floating-point format that JavaScript uses, leading to small precision errors. please check this.
    Note: I don’t want to use round, ceil, floor, etc in every form.
    This type of issue could be solved through the plugin backend (May be, I am not sure).

Viewing 1 replies (of 1 total)
  • Plugin Author codepeople

    (@codepeople)

    Hello @ramanandmehta

    Thank you very much for using our plugin. It is not a plugin issue or error but the Javascript engine implementation on browsers. Please look at the screenshot below. I evaluated the operation directly in the browser console:

    The plugin includes the PREC operation to fix the accuracy issue on the browser.

    PREC(X, Y) rounds the number X with Y decimals, and you should use it as the outermost operation in the equation.

    You can edit your equation as follows:

    PREC(0.0166666667 * 60, 2, true)

    It must be read as follows: Rounds the result of the mathematical operation 0.0166666667 * 60 with two decimal places, and the last parameter tells the plugin not to include the decimal places (.00) if the mathematical operation returns an integer number.

    Best regards.

Viewing 1 replies (of 1 total)
  • The topic ‘precision issues with floating-point’ is closed to new replies.