openglhierarchyclippingstencil-buffer

How to use stencil buffer to achieve hierarchical clipping


What I'm trying to achieve is illustrated on the image below:

illustration

Let's say we have stencil buffer in a state so only red section is filled at the moment. What actions do I need to perform when I update stencil buffer with a section which is marked yellow, so in the end only green section would be the final state of the stencil buffer?

I need this to achieve nested element content clipping, to prevent content of the element to be rendered beyond the boundaries of them both combined.

So far I have tried various boolean operations involving stencil test to no avail, which brought more confusion than any progress.

Please note that scissor test is not actual for this task, because elements may have arbitrary shape and rotation.


Solution

  • If you've a hierarchical structure,then you've layers. Each child is only allowed to draw in the region which is defined by its parent. So each layer has to count up the stencil buffer (GL_INCR) and is only allowed to draw, if the current value of the stencil buffer is equal to the layer "depth":

    int layer = 0;
    glStencilFunc(GL_ALWAYS, 1, 0xff);
    glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
    
    // draw initial layer
    // ...
    
    layer ++;
    glStencilFunc(GL_EQUAL, layer, 0xff);
    glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
    
    // draw layer 1
    // ...
    
    layer ++;
    glStencilFunc(GL_EQUAL, layer, 0xff);
    glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
    
    // draw layer 2
    // ...
    

    Note, in general the stencil buffer has 8 bits, this limits the number of layers to 256.