If I have a matrix then the point O(0,0,0) will be translated to some point P(x, y, z). So rotating a matrix about its current position is effectively multiplying the matrix by a rotation matrix about P.
So I want a function like:
mat4 rotate(mat4 matrix, vec3 axis, float angle);
my current idea is:
vec4 p = {0, 0, 0, 1};
p = p * matrix;
generate translation matrix T, from point p
generate rotation matrix R, from axis and angle
return matrix * T * R * -T;
but I feel like there should be a more efficient way to do this...
Yep, that's how I'd do it. But one subtle correction, reverse the order of -T and T:
return matrix * -T * R * T
You want to first 'undo' the translational origin of matrix, then rotate, then re-do translational origin. This is easier to see if you take, for example, a traditional scale/rotate/translate matrix (S * R2 * T), expand it, then you can see more easily:
(S * R2 * T) * -T * R * T
Is doing what you want.
EDIT: With respect to efficiency, totally depends on usage. No, this is not 'great' -- usually you have more information about matrix that will allow you to do this in a less round-about way. E.g., if the matrix is constructed from S * R * T above, obviously we could have simply changed the way that the matrix is constructed in the first place -- S * R2 * R * T, injecting the rotation where it should be without having to 'undo' anything.
But unless you are doing this in real-time on 10K+ matrix that need to be recomputed each time, then it should not be a problem.
If matrix is coming from an unknown source and you need to modify it ex-post-facto, indeed, there really isn't any other choice.