I'm searching for a way to determine if a camera (a) is looking at the same direction than a specific object (b).
I'm only working with the rotations because here we don't care about the positions (so we can consider both are located at the origin).
I'm working with quaternions, using gl-matrix.
After many searches and tests, I understood that I can determine the "distance" (magnitude ?) between the two quaternions using a dot product.
var a = quat.create();
quat.rotateY(a, a, Math.PI);
var b = quat.create();
quat.rotateY(b, b, Math.PI);
quat.dot(a, b)
That works well in many cases (it returns a number in range [0..1] : from 1 if both are exactly looking at the same direction to 0 if both are totally opposite).
However, in my case, I don't want to care about the "roll" between the objects. I mean that the camera (a) can be turned upside down, relatively to (b), but still looking at the same point.
For example, if I turn b by 180 degrees around Z, I get a dot product that is around 0, while it's still looking at the same direction.
var a = quat.create();
quat.rotateY(a, a, Math.PI);
var b = quat.create();
quat.rotateY(b, b, Math.PI);
quat.rotateZ(b, b, Math.PI);
quat.dot(a, b);
I tried many things, such as multiplying the inverse of a with b, or [s]lerp, but I still can't get anything that fits my requirements.
Of course, I can't simply separate and work with the absolute Z axis, because everything is all relative here, so the roll could be around any axis.
How can I get that result ?
EDIT : Thanks to LutzL answer, here's how I implemented the solution :
var r = quat.create();
quat.invert(r, a);
quat.multiply(r, r, b);
var distance = r[3]*r[3] - r[0]*r[0] - r[1]*r[1] + r[2]*r[2];
I interpret it the way that the unit vector of the Z axis (in local coordinates) is your forward pointing vector?
The quaternion a describes the translation of the local coordinate frame to the global coordinate frame. Thus a vector v in local coordinates has direction a*v*a' (a'=conjugate of a=inverse for unit quaternion) in the global frame. The direction of the Z axis thus changes according to ka=a*k*a'.
The Z direction of the second object is thus kb=b*k*b'. The cosine of the angle between them is the scalar product of ka and kb which is also the real part of ka'*kb=a*k'*a'*b*k*b'. The real part of any quaternion v remains unchanged in a'*v*a, so
scal(ka,kb) = real( ka * kb' )
= -real( k * a' * b * k * b' * a ) = -real( k * (a' * b) * k * (a'* b)' )
I do not know the specifics of the GL implementation, but the operations are to compute the product p1=a'*b, rotate p1 about the Z-axis by 180 degrees to p2, what in practice is just flipping the signs of the i and j coefficients, and forming the scalar product of p1 and p2. The short version is
scal(ka,kb) = p1.w*p1.w - p1.x*p1.x - p1.y*p1.y + p1.z*p1.z