androidopengl-esopengl-es-2.0android-camera2grafika

Camera2 OpenGL random glitches in video caused by Matrix.rotateM


I used the Grafika app and modified it to use the Camera2 API instead of the old API. I needed to use an OpenGL solution as I need to draw a watermark on top of the video, and Grafika was really useful. Unfortunately, my output videos are recording with random "flickers" of frames in the wrong orientation. I am looking to resolve the flickering issue, or at least understand why it is happening.

Originally, I managed to successfully record video, with sound, and drawing the watermark on top, but the video was in the wrong orientation as I required portrait videos. In order to achieve this, I used MediaMuxer.serOrientationHint() to configure the output file as portrait, and also applied a rotation to the transform matrix to ensure OpenGL frames were drawn in portrait, see below:

private void handleFrameAvailable(float[] transform, long timestampNanos) {

    mVideoEncoder.drainVideoEncoder(false);

    Matrix.rotateM(transform, 0, 270, 0, 0, 1); //Added these to rotate video frames
    Matrix.translateM(transform, 0, -1, 0, 0); //Added these to rotate video frames

    mFullScreen.drawFrame(mTextureId, transform);

    //...drawing of watermark happens here...//

    if (VERBOSE) { Log.e(TAG,"HandleVideo: "+timestampNanos); }
    mInputWindowSurface.setPresentationTime(timestampNanos);
    mInputWindowSurface.swapBuffers();
}

See below a regular frame and a glitched frame. In a 5 second video, around 20-30 non-consecutive frames may be like this.

good frame bad frame


Solution

  • If you aren't resetting transform to identity matrix then you are accumulating transforms on each frame. Try:

    Matrix.setIndentityM(transform, m);
    

    before apply translation and rotation.

    Besides, could be orientation:

    if (AppSetting.getValue(activity, Config.ORIENTATION, "").equalsIgnoreCase("Portrait")) {
        Matrix.rotateM(mTmpMatrix, 0, 270, 0, 0, 1);
        Matrix.translateM(mTmpMatrix, 0, -1, 0, 0);
    }