How to use a stencil buffer mask in ogre?

How to use the stencil buffer to create a mask in OGRE?

That is, some objects should be rendered first into the stencil buffer and generate a mask (say 0 is background, 0xFF is foreground); then render the scene itself, using the stencil buffer as a mask so only pixels where it is 0xFF are rendered.

I guess I should use a RenderQueueListener, but I can't make it work. Here's what I'm doing right now:

void StencilOpQueueListener::renderQueueStarted(Ogre::uint8 queueGroupId, const Ogre::String& invocation, bool& skipThisInvocation) {
   if (queueGroupId == Ogre::RENDER_QUEUE_1) {
      Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
      rs->clearFrameBuffer(Ogre::FBT_STENCIL, Ogre::ColourValue::Black, 1.0, 0xFF);
      rs->setStencilCheckEnabled(true);
      rs->_setColourBufferWriteEnabled(false, false, false, false);
      rs->setStencilBufferParams(
         Ogre::CMPF_ALWAYS_PASS, // compare
         0x1, // refvalue
         0xFFFFFFFF, // mask
         Ogre::SOP_REPLACE, Ogre::SOP_REPLACE, // stencil fail, depth fail
         Ogre::SOP_REPLACE, // stencil pass + depth pass
         false); // two-sided operation? no
   }
}

void StencilOpQueueListener::renderQueueEnded(Ogre::uint8 queueGroupId, const Ogre::String& invocation, bool& repeatThisInvocation)  {
   if (queueGroupId == Ogre::RENDER_QUEUE_1)  {
      Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
      rs->setStencilCheckEnabled(false);
      rs->setStencilBufferParams();
   }
}

And I set the entities that should render to the stencil buffer are set with:

   entity->setRenderQueueGroup(RENDER_QUEUE_1);

What am I doing wrong? Any examples of how this is done in Ogre? Thanks!

For reference, this is how I do it in pure OpenGL:

/* Enable stencil test and leave it enabled throughout */
glClearStencil(0xFF);
glClearColor(1, 1, 1, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0xFFFFFFFF);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

// render into the stencil buffer. This should render only the selector objects
renderStencil();

// restore the rendering
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);

// only render "inside" the silhouette -- we want the overlap
glStencilFunc(GL_EQUAL, 0x1, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

render(); // now we render objects against that mask

Solution

you should synthesize the second step as well, which you mentioned in

// only render "inside" the silhouette -- we want the overlap

glStencilFunc(GL_EQUAL, 0x1, 0xFFFFFFFF); glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

This means, you should apply the mask you set in RENDER_QUEUE_1 to the rendered scene, like you do in the case of gl calls. The main rendering takes place in Ogre's RENDER_QUEUE_MAIN by default, thus you should add this to your renderQueueStarted method:

if (queueGroupId == Ogre::RENDER_QUEUE_MAIN)
{
    Ogre::RenderSystem * rs = Ogre::Root::getSingleton().getRenderSystem();
    rs->_setColourBufferWriteEnabled(true, true, true, true);
    rs->setStencilCheckEnabled(true);
    rs->setStencilBufferParams(Ogre::CMPF_EQUAL,0x1,0xFFFFFFFF,
        Ogre::SOP_KEEP,Ogre::SOP_KEEP,Ogre::SOP_KEEP,false);     
}

also, switch off the stenciling in renderQueueEnded only after main rendering is finished:

if ( queueGroupId == Ogre::RENDER_QUEUE_MAIN )
{
    Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
    rs->setStencilCheckEnabled(false);
    rs->setStencilBufferParams();
}