javaandroidarcoreandroid-sensorseuler-angles

How can i get the Yaw Pitch and Roll values from the Camera Pose of the ARCore?


I have this Java code using the ARCore Pose :

private void updateCameraPose(Pose pose) {

        //float[] quaternion = new float[4];
        //pose.getRotationQuaternion(quaternion, 0);

        float x = pose.qx(); //quaternion[0];
        float y = pose.qy(); //quaternion[2];
        float z = pose.qz(); //quaternion[1];
        float w = pose.qw(); //quaternion[3];

        double yaw = atan2(2.0 * (w * y + x * z), 1.0 - 2.0 * (y * y + z * z));
        double pitch = asin(2.0 * (w * x - y * z));
        double roll = atan2(2.0 * (w * z + x * y), 1.0 - 2.0 * (x * x + z * z));

    }

When i try to validate these values by orientating the phone in each axis, it doesn't match at all the behavior that i can see here in the simulator.

I need to get the Yaw / Pitch and Roll values of my Android phone at any time and i am not sure either to get it correctly from the ARCore or calculate them correctly (with a correct mathematical formula).

I guess that the Rotation matrix from the ARCore doesn't especially match the one in the simulator in the first place, but that's fine i can map afterwards when orientating the phone (Maybe the ARCore Yaw is the Pitch in the simulator).

But to do that i need first to be sure the maths are correct.


Solution

  • ARCamera's Transform Matrix Decomposition

    Below you can find a solution for using a 4x4 transform matrix in Google ARCore 1.48. The basic idea is quite simple: I used displayOrientedPose instance property to get a virtual camera's pose (for a default portrait screen orientation) in world space for the latest frame. The directions of the axes are as follows: +X points right, +Y points up, and -Z points in the direction the AR camera is looking (i.e. north).

    I my case, when using the pose property instead of the displayOrientedPose, the zero Z-axis value is only achieved when the smartphone is rotated to landscape right orientation (90 CCW).

    The initial orientation [0,0,0] is recorded at the moment when your AR session is run (usually within couple of seconds after the app is launched). At this moment, do not hold the phone at an angle relative to any axis.

    Then I filled the empty matrix with 16 values ​​using the toMatrix() method. The order of the matrix indices is as follows:

    ┌               ┐
    |  0  4  8  12  |
    |  1  5  9  13  |
    |  2  6  10 14  |
    |  3  7  11 15  |
    └               ┘
    

    The formulas for matrix decomposition can be easily found on the web. Every of 9 top-left elements of the 4x4 transform matrix is ​​stored in radians, so we use the toDegrees() method here to convert to degrees. This way you get the correct Pitch (x), Yaw (y) and Roll (z) euler angles. Negative values ​​are obtained when rotating the camera clockwise (CW) around any of the axes.

    enter image description here

    Here is my code (sorry, it's in Kotlin):

    private fun matrix(fragment: ArFragment) {
        val matrix = FloatArray(16)
            
        val arFrame = fragment.arSceneView.arFrame
        arFrame?.camera?.displayOrientedPose?.toMatrix(matrix, 0)
    
        val xRad = asin(-matrix[9])                 // arcsin * -1
        val yRad = atan2(matrix[8], matrix[10])     // 2-args arctan
        val zRad = atan2(matrix[1], matrix[5])      // 2-args arctan
    
        val xDeg = truncate(this.toDegrees(xRad)).toInt()
        val yDeg = truncate(this.toDegrees(yRad)).toInt()
        val zDeg = truncate(this.toDegrees(zRad)).toInt()
    
        Toast.makeText(
            this@MainActivity,
            "x: ${xDeg}, y: ${yDeg}, z: ${zDeg}",
            Toast.LENGTH_SHORT
        ).show()
    }
    
    private fun toDegrees(radians: Float): Float {
        return radians * 180 / PI.toFloat()
    }