I have my own 3D engine implementation around OpenGL (C++) and it has worked fine for everything these last years.
But today I stumbled upon a problem. I have this scene with spheres (planets around a sun, orbit rings and things like that, very very simple) and I want to ray pick them with the mouse.
As long as the camera/view matrix is identity, picking works. When the camera is rotated and then moved, the picking goes completely haywire. I have been searching for a solution for a while now so now I'm asking you guys.
This is the code (summarized for this question):
mat4f mProj = createPerspective(PI / 4.0f, float(res.x) / float(res.y), 0.1f, 100.0f);
mat4f mCamera = createTranslation(-1.5f, 3, -34.0f) * createRotationZ(20.0f * PI / 180.0f) * createRotationX(20.0f * PI / 180.0f);
... render scene, using shaders that transform vertices with gl_Position = mProj * mCamera * aPosition;
mat4f mUnproject = (mProj * mCamera).getInverse();
vec2f mouseClip(2.0f * float(coord.x) / float(res.x) - 1.0f, 1.0f - 2.0f * float(coord.y) / float(res.y));
vec3f rayOrigin = (mUnproject * vec4f(mouseClip, 0, 1)).xyz();
vec3f rayDir = (mUnproject * vec4f(mouseClip, -1, 1)).xyz();
// In a loop over all planets:
mat4f mObject = createRotationY(planet.angle) * createTranslation(planet.distance, 0, 0);
vec3f planetPos = mObject.transformCoord(vec3f(0, 0, 0));
float R = planet.distance;
float a = rayDir.dot(rayDir);
float b = 2 * rayDir.x * (rayOrigin.x - planetPos.x) + 2 * rayDir.y * (rayOrigin.y - planetPos.y) + 2 * rayDir.z * (rayOrigin.z - planetPos.z);
float c = planetPos.dot(planetPos) + rayOrigin.dot(rayOrigin) -2 * planetPos.dot(rayOrigin) - R * R;
float d = b * b - 4 * a * c;
if (d >= 0)
HIT!
So when I use identity for mCamera, everything works fine, even when I use only rotation for mCamera, it works fine. It is when I start using the translation that it goes completely wrong. Anyone knows where I am going wrong?
BDL's answer was spot on and put me back in the right direction. Indeed, when transforming coordinates myself, I forgot to do the perspective-divide. After writing so much shader code where the gpu does this for you, you forget about these things.
It is logical that this only gave issues when the camera moved and not when it was at (0, 0, 0) as then, the translation part of the transformation matrices stayed 0 and the w-factor of the coordinates were unaffected.
I immediately wrote transformCoord and transformNormal implementations in my matrix classes to prevent this error from happening again.
Also, the ray origin and direction were incorrect, although I don't really understand why yet. I now take the origin from my camera matrix (inverted of course) and calculate the direction the same way but now subtract the camera position from it to make it a direction vector. I normalize it, although I don't think it is really necessary in this case, but normalizing it will make its numbers look more readable when debugging anyway.
This works:
vec2f mouseClip(2.0f * float(coord.x) / float(res.x) - 1.0f, 1.0f - 2.0f * float(coord.y) / float(res.y));
mat4f mUnproject = (mProj * mCamera).getInverse();
mat4f mInvCamera = mCamera.getInverse();
vec3f rayOrigin(mInvCamera.m[12], mInvCamera.m[13], mInvCamera.m[14]);
vec3f rayDir = (mUnproject.transformCoord(vec3f(mouseClip, 1)) - rayOrigin).normalized();
... per planet
vec3f planetPos = mObject.transformCoord(vec3f(0, 0, 0));
float a = rayDir.dot(rayDir);
float b = 2 * rayDir.x * (rayOrigin.x - planetPos.x) + 2 * rayDir.y * (rayOrigin.y - planetPos.y) + 2 * rayDir.z * (rayOrigin.z - planetPos.z);
float c = planetPos.dot(planetPos) + rayOrigin.dot(rayOrigin) -2 * planetPos.dot(rayOrigin) - 0.4f * 0.4f;
float d = b * b - 4 * a * c;
if (d >= 0)
... HIT!