stencil buffer for deferred shading with open scene graph

I am trying to implement Deferred Shading according to the tutorials 35-37 on http://ogldev.atspace.co.uk/www/tutorial35/tutorial35.html with Open Scene Graph 3.4.0

Right now, I'm struggling to get the stencil pass correctly, so I simplified my code as far as possible. As a result, right now I am using three cameras:

• one RTT camera for postion-, normal- and color texture as well as writing the depth buffer.
• one stencil camera (set up as a RTT camera with no color output in the fragment shader) for writing into the stencil buffer using the depth test results
• one point light camera (set up as a HUD camera), using deferred shading and set to write only where the stencil buffer is not 0.

When I try to display the depth buffer with the point light camera it works. When I set the clear mask of pointLightCamera to STENCIL_BUFFER_BIT and set the stencil clear to 1, it displays everything. It displays nothing when setting it to 0. When I disable the clear settings in the pointLightCamera (as they should be) and enable any clear settings for the stencil buffer in the stencil camera, it has no effect whatsoever. The stencilFunc and stencilOperation are set as they are in the tutorials.

When I start the program it won't display anything (so all the stencil values are 0). This leads me to the conclusion, that my stencilCamera doesnt actually write into the stencil buffer, yet I have no idea why, I couldn't find anymore help with the osg examples or internet forums and I have tried everything I could think of.

Here are my set ups for the cameras and viewer (the rttCamera has my scene groupnode as a child, which contains a model, the stencilCamera has a sphere model as a child for the light volume and my pointLightCamera has a screenQuad as a child).

texDepth->setTextureSize(1024, 1024);
texDepth->setInternalFormat(GL_DEPTH24_STENCIL8_EXT);
texDepth->setSourceFormat(GL_DEPTH_STENCIL_EXT);
texDepth->setSourceType(GL_UNSIGNED_INT_24_8_EXT);

osg::ref_ptr<osg::Texture2D> texColor = createTexture();
osg::ref_ptr<osg::Texture2D> texPosition = createTexture();
osg::ref_ptr<osg::Texture2D> texNormal = createTexture();

//1. pass camera and set up
osg::ref_ptr<osg::Camera> rttCamera = createRTTCamera(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, texDepth, false);

rttCamera->setRenderOrder(osg::Camera::PRE_RENDER, 0);
rttCamera->attach(osg::Camera::COLOR_BUFFER0, texColor);
rttCamera->attach(osg::Camera::COLOR_BUFFER1, texPosition);
rttCamera->attach(osg::Camera::COLOR_BUFFER2, texNormal);
rttCamera->setClearColor(osg::Vec4(0.0, 0.0, 0.0, 0.0));

osg::ref_ptr<osg::Stencil> rttStencil = new osg::Stencil;
rttStencil->setWriteMask(0);
rttCamera->getOrCreateStateSet()->setAttribute(rttStencil, osg::StateAttribute::ON);

rttCamera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

osg::ref_ptr<osg::StateSet> ss = rttCamera->getOrCreateStateSet();
osg::ref_ptr<osg::Program> rttProg = new osg::Program;
osg::Shader* vertShader = osgDB::readShaderFile("pass1.vert");
osg::Shader* fragShader = osgDB::readShaderFile("pass1.frag");
rttProg->addShader(vertShader);
rttProg->addShader(fragShader);
ss->setAttributeAndModes(rttProg.get(), osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

osg::ref_ptr<osg::BlendFunc> bf = new osg::BlendFunc;
bf->setFunction(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
ss->setAttributeAndModes(bf, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

rttCamera->addChild(scene.get());

//2. pass: stencil pass camera and set up
osg::ref_ptr<osg::Camera> stencilCamera = createRTTCamera(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, texDepth, false);//createHUDCamera(0.0, 1.0, 0.0, 1.0);//
stencilCamera->setRenderOrder(osg::Camera::PRE_RENDER, 2);

stencilCamera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, texDepth);//depth buffer was filled by rttCamera
stencilCamera->getOrCreateStateSet()->setMode(GL_STENCIL_TEST, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);
stencilCamera->getOrCreateStateSet()->setMode(GL_DEPTH_TEST, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);  //depth test result will fill stencil buffer
osg::ref_ptr<osg::Depth> depth = new osg::Depth;
depth->setWriteMask(false);         //depth test is needed to compare scene to light volume, but light volume must not write into depth buffer
stencilCamera->getOrCreateStateSet()->setAttribute(depth, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

stencilCamera->getOrCreateStateSet()->setMode(GL_CULL_FACE, osg::StateAttribute::OFF | osg::StateAttribute::OVERRIDE);
stencilCamera->setClearMask(GL_STENCIL_BUFFER_BIT);

osg::ref_ptr<osg::StencilTwoSided> stencilWrite = new osg::StencilTwoSided;
stencilWrite->setFunction(osg::StencilTwoSided::FRONT, osg::StencilTwoSided::ALWAYS, 0, 0);
stencilWrite->setFunction(osg::StencilTwoSided::BACK, osg::StencilTwoSided::ALWAYS, 0, 0);
stencilWrite->setOperation(osg::StencilTwoSided::FRONT, osg::StencilTwoSided::KEEP, osg::StencilTwoSided::DECR_WRAP, osg::StencilTwoSided::KEEP);
stencilWrite->setOperation(osg::StencilTwoSided::BACK, osg::StencilTwoSided::KEEP, osg::StencilTwoSided::INCR_WRAP, osg::StencilTwoSided::KEEP);

stencilWrite->setWriteMask(osg::StencilTwoSided::FRONT, 0xFF);  //may not be needed
stencilWrite->setWriteMask(osg::StencilTwoSided::BACK, 0xFF);
stencilCamera->getOrCreateStateSet()->setAttribute(stencilWrite, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

stencilCamera->addChild(mtSphere);//lights.get());

osg::ref_ptr<osg::Program> stencilProg = new osg::Program;
stencilProg->addShader(osgDB::readShaderFile("nullTechnique.vert"));
stencilProg->addShader(osgDB::readShaderFile("nullTechnique.frag"));

stencilCamera->getOrCreateStateSet()->setAttributeAndModes(stencilProg.get(), osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

//3. pass: point light pass camera and set up
osg::ref_ptr<osg::Camera> pointLightCamera = createHUDCamera(0.0, 1.0, 0.0, 1.0);

pointLightCamera->setClearMask(0);
pointLightCamera->setRenderOrder(osg::Camera::POST_RENDER, 1);//PRE_RENDER, 2);
pointLightCamera->addChild(screenQuad);
ss = pointLightCamera->getOrCreateStateSet();

osg::ref_ptr<osg::Stencil> stencilRead = new osg::Stencil;
stencilRead->setFunction(osg::Stencil::NOTEQUAL, 1, 0xFF);  //render only where stencil buffer is != 0 (this will be 1 as set in stencil pass)
stencilRead->setWriteMask(0);//it should not write into the stencil buffer it reads from
ss->setAttribute(stencilRead, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);

osg::ref_ptr<osg::Depth> depthRead = new osg::Depth;
depth->setWriteMask(false);
ss->setAttribute(depth, osg::StateAttribute::ON);

pointLightCamera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, texDepth);
ss->setMode(GL_STENCIL_TEST, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);
ss->setMode(GL_DEPTH_TEST, osg::StateAttribute::OFF);

ss->setMode(GL_BLEND, osg::StateAttribute::ON);         //all light passes shall add their renderings to the previous light passes
osg::ref_ptr<osg::BlendFunc> renderAddBlendFunc = new osg::BlendFunc;
renderAddBlendFunc->setFunction(GL_ONE, GL_ONE);
ss->setAttribute(renderAddBlendFunc, osg::StateAttribute::ON);
osg::ref_ptr<osg::BlendEquation> renderAddBlendEq = new osg::BlendEquation;
renderAddBlendEq->setEquation(osg::BlendEquation::FUNC_ADD);

ss->setAttribute(renderAddBlendEq, osg::StateAttribute::ON);
osg::ref_ptr<osg::CullFace> cullFacePointLightPass = new osg::CullFace(osg::CullFace::FRONT);

osg::ref_ptr<osg::Program> pointLightProg = new osg::Program;
vertShader = osgDB::readShaderFile("pass2.vert");
fragShader = osgDB::readShaderFile("pass2.frag");
pointLightProg->addShader(vertShader);
pointLightProg->addShader(fragShader);
ss->setAttributeAndModes(pointLightProg.get(), osg::StateAttribute::ON);
ss->setTextureAttributeAndModes(0, texColor);
ss->setTextureAttributeAndModes(1, texPosition);
ss->setTextureAttributeAndModes(2, texNormal);
ss->setTextureAttributeAndModes(3, texDepth);

ss->addUniform(new osg::Uniform("tDiffuse", 0));
ss->addUniform(new osg::Uniform("tPosition", 1));
ss->addUniform(new osg::Uniform("tNormals", 2));
ss->addUniform(new osg::Uniform("tDepth", 3));

ss->addUniform(new osg::Uniform("lightPosition", osg::Vec3(0.0, 0.0, 0.0)));

osg::Vec3 eye, center, up;
rttCamera->getViewMatrixAsLookAt(eye, center, up);
ss->addUniform(new osg::Uniform("cameraPosition", eye));

pointLightCamera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER);

osg::ref_ptr<osg::Group> root = new osg::Group;
root->addChild(rttCamera);
root->addChild(stencilCamera);
root->addChild(pointLightCamera);

osgViewer::Viewer viewer;
viewer.setCameraManipulator(new osgGA::TrackballManipulator);
viewer.getCamera()->setComputeNearFarMode(osg::CullSettings::DO_NOT_COMPUTE_NEAR_FAR);
viewer.setSceneData(root.get());

osg::DisplaySettings::instance()->setMinimumNumStencilBits(8);

(createRTTCamera and createHUDCamera are taking from the OSG Cookbook).

Okay, I finally found the problem.

I was using FRAME_BUFFER_OBJECT as my render target implementation for the RTT cameras, but FRAME_BUFFER for my HUD camera. I later noticed I got an OpenGL "invalid value" error with this. Now that I use FRAME_BUFFER_OBJECT for all three, I can share the stencil values and it works. I just needed to write the results of my former HUD camera into a new texture, which I then read with a last camera that was a new HUD Camera and that had FRAME_BUFFER as the render target.