Import transformation matrix from different coordinates system

I'm currently running two game engines in parallel that have 2 distinct coordinate systems:

• Tomb Raider (TR) coordinate system;
• Super Mario 64 (SM64) coordinate system;

To transform a vertex position (TR->SM64) I simply do:

• x' = x/scaling
• y' = -y/scaling
• z' = -z/scaling

The inverse transform (SM64 -> TR) is obviously similar:

• x' = x*scaling
• y' = -y*scaling
• z' = -z*scaling

Now I want to import the animation transformation matrices from SM64 into TR to apply Mario's animation to Lara. Here the problem becomes the rotations quaternions.

SM64 animation system applies a base transformation to the character and then keeps stacking more transformation matrices with translation and rotation for each bone of the character rig.

To fix the signage and scaling of the position I multiplied the base transformation matrix by (S, -S, -S) where S is the scaling factor. That fixed the position of the bones in when transposed to TR but the rotations quaternions are completely broken.

To try to fix it I applied some rotations to the transformation matrix:

mat = mat4(
    originalMatrix[0][0], originalMatrix[0][1], originalMatrix[0][2], originalMatrix[0][3], 
    originalMatrix[1][0], originalMatrix[1][1], originalMatrix[1][2], originalMatrix[1][3], 
    originalMatrix[2][0], originalMatrix[2][1], originalMatrix[2][2], originalMatrix[2][3],
    originalMatrix[3][0], originalMatrix[3][1], originalMatrix[3][2], originalMatrix[3][3]
);

mat.rotateX(-M_PI/2);
mat.rotateZ(M_PI/2);

position=mat.getPos();
rot=mat.getRot(); //gets a rotation quaternion

But unfortunately it flips Y global angle and for more complex bone rotations (like the feet) it becomes a complete mess:

https://i.imgur.com/A00clc3.mp4

I truly don't know how to properly transform a transformation matrix from one coordinates system to another.

I saw several video tutorials about how to change coordinate systems but I still couldn't manage it.

EDIT: One of the examples I saw and was suggested again was to do

Tinv * Matrix * T

but unfortunately that didn't work well:

mat4 T = mat4(
     1.0f,  0.0f,  0.0f,  0.0f,
     0.0f, -1.0f,  0.0f,  0.0f,
     0.0f,  0.0f, -1.0f,  0.0f,
     0.0f,  0.0f,  0.0f,  1.0f
);

mat = T.inverse()*mat*T;

The position gets inverted bellow the ground (I removed the signage hack in the scaling to avoid mayhem) and the rotations remain wrong.

With the help of Open Lara developer XProger I managed to find the solution for this problem.

For starts lets define T as the transformation matrix between the two coordinate systems:

mat4 T = mat4(
     1.0f,  0.0f,  0.0f,  0.0f,
     0.0f, -1.0f,  0.0f,  0.0f,
     0.0f,  0.0f, -1.0f,  0.0f,
     0.0f,  0.0f,  0.0f,  1.0f
);

1 - I multiplied all vertices from the body part I wanted to manipulate at TR side for T so they are in the same coordinates as SM64:

vertex = T*vertex;

2 - I rotated the SM64 matrix so It lined up with TR coordinates:

mat = mat4(
    originalMatrix[0][0], originalMatrix[0][1], originalMatrix[0][2], originalMatrix[0][3], 
    originalMatrix[1][0], originalMatrix[1][1], originalMatrix[1][2], originalMatrix[1][3], 
    originalMatrix[2][0], originalMatrix[2][1], originalMatrix[2][2], originalMatrix[2][3],
    originalMatrix[3][0], originalMatrix[3][1], originalMatrix[3][2], originalMatrix[3][3]
);

mat.rotateX(M_PI/2);
mat.rotateZ(-M_PI/2);

3 - I transformed the matrix coordinates system:

mat = T.inverse()*mat*T;

4 - Changed the scaling I was applying at the beginning of the SM64 transforms from [S,-S,-S] to [S,S,S].

After all of this I finally have the animation transformation matrices correctly transposed from SM64 engine into TR.