[SOLVED] opengl ray picking/tracing with cardboard (GVR)

opengl ray picking/tracing with cardboard (GVR)

For the demonstration, I made a small application that draws three squares of random size and random location. Drawing squares:

private void renderObject(float[] modelViewProjection, FloatBuffer vertexCoordsBuf, int i) {
        GLES20.glUseProgram(mProgram);
        GLES20.glEnableVertexAttribArray(mPositionHandle);
        GLES20.glVertexAttribPointer(
                mPositionHandle, 3,
                GLES20.GL_FLOAT, false,
                12, vertexCoordsBuf);


        GLES20.glUniform4fv(mColorHandle,
                1, colors, i * 4);

        GLES20.glUniformMatrix4fv(mMVPMatrixHandle,
                1, false, modelViewProjection, 0);

        // Draw the square
        GLES20.glDrawElements(
                GLES20.GL_TRIANGLES, 6,
                GLES20.GL_UNSIGNED_SHORT, indexesBuffer);
        // Disable vertex array
        GLES20.glDisableVertexAttribArray(mPositionHandle);
}

I also display the cursor (white small square) in the center of the screen using the orthogonal projection:

private void renderCursor(Eye eye) {
        GLES20.glDepthFunc(GLES20.GL_ALWAYS);
        GLES20.glDepthMask(false);

        GLES20.glUseProgram(mProgram);
        GLES20.glEnableVertexAttribArray(mPositionHandle);
        GLES20.glVertexAttribPointer(
                mPositionHandle, 3,
                GLES20.GL_FLOAT, false,
                12, cursorVertexCoordsBuf);


        GLES20.glUniform4fv(mColorHandle,
                1, colors, 12);

        float[] perspectiveMatrix = eye.getPerspective(Z_NEAR, Z_FAR);
        float[] orthoViewMatrix = new float[16];
        float[] modelViewMatrix = new float[16];
        Matrix.orthoM(orthoViewMatrix, 0, -1, 1, -1, 1, Z_NEAR, Z_FAR);
        Matrix.multiplyMM(modelViewMatrix, 0, perspectiveMatrix, 0, orthoViewMatrix, 0);

        GLES20.glUniformMatrix4fv(mMVPMatrixHandle,
                1, false, modelViewMatrix, 0);


        // Draw the square
        GLES20.glDrawElements(
                GLES20.GL_TRIANGLES, 6,
                GLES20.GL_UNSIGNED_SHORT, indexesBuffer);
        // Disable vertex array
        GLES20.glDisableVertexAttribArray(mPositionHandle);
        GLES20.glDepthFunc(GLES20.GL_LEQUAL);
        GLES20.glDepthMask(true);
    }

For clarity in the center of the visible area for stereo mode, I draw red dots. To see how the cursor deviates from the center of the screen.

How it see in stereo mode. How you can see white cursor moved from centers of screens:

If I have disabled the stereo mode then everything works fine. I use a method to determine which object in the center of the screen I'm producing ray tracing for screenWidth/2 and screenHeight/2 coordinates:

public boolean rayPicking(int viewWidth, int viewHeight, float rx, float ry,
                              float[] viewMatrix, float[] projMatrix, float[] modelViewMatrix,
                              float[] objectCoords, short[] objectIndexes) {
        float[] near_xyz = unProject(rx, ry, 0, viewMatrix, projMatrix, viewWidth, viewHeight);
        float[] far_xyz = unProject(rx, ry, 1, viewMatrix, projMatrix, viewWidth, viewHeight);
        int coordCount = objectCoords.length;
        float[] convertedSquare = new float[coordCount];
        float[] resultVector = new float[4];
        float[] inputVector = new float[4];
        for (int i = 0; i < coordCount; i = i + 3) {
            inputVector[0] = objectCoords[i];
            inputVector[1] = objectCoords[i + 1];
            inputVector[2] = objectCoords[i + 2];
            inputVector[3] = 1;
            Matrix.multiplyMV(resultVector, 0, modelViewMatrix, 0, inputVector, 0);
            convertedSquare[i] = resultVector[0] / resultVector[3];
            convertedSquare[i + 1] = resultVector[1] / resultVector[3];
            convertedSquare[i + 2] = resultVector[2] / resultVector[3];
        }


        ArrayList<Triangle> triangles = new ArrayList<>();
        for (int i = 0; i < objectIndexes.length; i = i + 3) {
            int i1 = objectIndexes[i] * 3;
            int i2 = objectIndexes[i + 1] * 3;
            int i3 = objectIndexes[i + 2] * 3;
            triangles.add(
                    new Triangle(
                            new float[]{
                                    convertedSquare[i1],
                                    convertedSquare[i1 + 1],
                                    convertedSquare[i1 + 2]
                            },
                            new float[]{
                                    convertedSquare[i2],
                                    convertedSquare[i2 + 1],
                                    convertedSquare[i2 + 2]
                            },
                            new float[]{
                                    convertedSquare[i3],
                                    convertedSquare[i3 + 1],
                                    convertedSquare[i3 + 2]
                            }
                    )
            );
        }

        for (Triangle t : triangles) {
            float[] point = new float[3];
            int intersects = Triangle.intersectRayAndTriangle(near_xyz, far_xyz, t, point);
            if (intersects == 1 || intersects == 2) {
                return true;
            }
        }
        return false;
}

But if i turn on the stereo mode, the cursor is not displayed in the center of the screen. And for the left eye is moved in the left from the center of the visible area of the left eye, and for the right is appropriately to the right. And in this case, I do not understand how to get the real screen coordinates for the cursor so that it can produce ray tracing? I will be glad to any prompts ;)

Full worked sample you can get from github: https://github.com/Anton111111/AndroidGVRRayPickingSample

And all placed in one file to easy read it: https://github.com/Anton111111/AndroidGVRRayPickingSample/blob/master/app/src/main/java/com/anton111111/ray_picking/MainActivity.java

P.s. Please, if you want to set minus to my question simple add comment to explain what's wrong with my question.

Solution