I am using a stencil mask on a 3d object using the hello ar java demo however i am running into some unexpected behaviour. My stencil mask correctly occludes the plane renderer but the 3d object (andy) does not seem to react expectedly. Instead he seems to get flipped as shown in the picture. I am not sure how to approach fixing this issue. Attached is the code snippet doing the stencil masking
Image of stencil correctly working on plane buffer but failing on 3d model
GLES20.glClear ( GLES20.GL_STENCIL_BUFFER_BIT );
GLES20.glEnable(GLES20.GL_STENCIL_TEST);
GLES20.glColorMask(false, false, false, false);
GLES20.glDepthMask(false);
GLES20.glStencilFunc(GLES20.GL_NEVER, 1, 0xFF);
GLES20.glStencilOp(GLES20.GL_REPLACE, GLES20.GL_KEEP, GLES20.GL_KEEP);
GLES20.glStencilMask(0xFF);
GLES20.glClear(GLES20.GL_STENCIL_BUFFER_BIT);
// controls how pixels are rendered in the stencil mask
quadDrawer.draw();
GLES20.glColorMask(true, true, true, true);
GLES20.glDepthMask(true);
GLES20.glStencilMask(0xFF);
GLES20.glStencilFunc(GLES20.GL_EQUAL, 0, 0xFF);
// Visualize planes.
// reacts correctly to the stencil mask
planeRenderer.drawPlanes(
session.getAllTrackables(Plane.class), camera.getDisplayOrientedPose(), projmtx);
// Visualize anchors created by touch.
float scaleFactor = 1.0f;
for (ColoredAnchor coloredAnchor : anchors) {
if (coloredAnchor.anchor.getTrackingState() != TrackingState.TRACKING) {
continue;
}
// Get the current pose of an Anchor in world space. The Anchor pose is updated
// during calls to session.update() as ARCore refines its estimate of the world.
coloredAnchor.anchor.getPose().toMatrix(anchorMatrix, 0);
// Update and draw the model and its shadow.
// does not react correctly to the
virtualObject.updateModelMatrix(anchorMatrix, scaleFactor);
virtualObjectShadow.updateModelMatrix(anchorMatrix, scaleFactor);
virtualObject.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
virtualObjectShadow.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
}
GLES20.glDisable(GLES20.GL_STENCIL_TEST);
Forgot to update the "solution" i found for this. Instead of applying the stencil mask over the 3d object, i render the background again but passing it through the stencil mask. This means that the background will overlay the 3d object hence achieving the desired "masking" effect.