I'm working on a vector class. While testing the vector class, I ran into a failed test case when comparing the magnitude of a unit vector with 1.
When comparing them, they both appear to be 1, so what would cause this test to fail?
I have cut out as much as possible to get down to the root cause of the problem. I could just cast the numbers to strings and compare those, but that would only fix the test case, allowing the problem to show up again later down the line. I'm using Lua 5.1 interpreter (to identify the root cause), bundled with LuaRocks.
local v = 0.70710678118655^2 + 0.70710678118655^2
print(v, v == 1)
v == 1 should be true, not false.
When converting a float to a string, Lua rounds off to only 15 significant digits. Try making it 17 digits to get an exact representation.
local v = 0.70710678118655^2 + 0.70710678118655^2
print(("%.17g"):format(v), v == 1)
Output:
1.0000000000000071 false