androidopengl-esarcorestencil-buffer

Problem using stencil mask on 3d object in arcore


I am using a stencil mask on a 3d object using the hello ar java demo however i am running into some unexpected behaviour. My stencil mask correctly occludes the plane renderer but the 3d object (andy) does not seem to react expectedly. Instead he seems to get flipped as shown in the picture. I am not sure how to approach fixing this issue. Attached is the code snippet doing the stencil masking

Image of stencil correctly working on plane buffer but failing on 3d model

  GLES20.glClear ( GLES20.GL_STENCIL_BUFFER_BIT );
  GLES20.glEnable(GLES20.GL_STENCIL_TEST);
  GLES20.glColorMask(false, false, false, false);
  GLES20.glDepthMask(false);
  GLES20.glStencilFunc(GLES20.GL_NEVER, 1, 0xFF);
  GLES20.glStencilOp(GLES20.GL_REPLACE, GLES20.GL_KEEP, GLES20.GL_KEEP);
  GLES20.glStencilMask(0xFF);
  GLES20.glClear(GLES20.GL_STENCIL_BUFFER_BIT);

  // controls how pixels are rendered in the stencil mask
  quadDrawer.draw();

  GLES20.glColorMask(true, true, true, true);
  GLES20.glDepthMask(true);
  GLES20.glStencilMask(0xFF);
  GLES20.glStencilFunc(GLES20.GL_EQUAL, 0, 0xFF);

  // Visualize planes.
  // reacts correctly to the stencil mask
  planeRenderer.drawPlanes(
      session.getAllTrackables(Plane.class), camera.getDisplayOrientedPose(), projmtx);

  // Visualize anchors created by touch.
  float scaleFactor = 1.0f;

  for (ColoredAnchor coloredAnchor : anchors) {
    if (coloredAnchor.anchor.getTrackingState() != TrackingState.TRACKING) {
      continue;
    }
    // Get the current pose of an Anchor in world space. The Anchor pose is updated
    // during calls to session.update() as ARCore refines its estimate of the world.
    coloredAnchor.anchor.getPose().toMatrix(anchorMatrix, 0);

    // Update and draw the model and its shadow.
    // does not react correctly to the 
    virtualObject.updateModelMatrix(anchorMatrix, scaleFactor);
    virtualObjectShadow.updateModelMatrix(anchorMatrix, scaleFactor);
    virtualObject.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
    virtualObjectShadow.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
  }

  GLES20.glDisable(GLES20.GL_STENCIL_TEST);

Solution

  • Forgot to update the "solution" i found for this. Instead of applying the stencil mask over the 3d object, i render the background again but passing it through the stencil mask. This means that the background will overlay the 3d object hence achieving the desired "masking" effect.