openglintersectionblendingglblendfunc

OpenGL Blending - fill intersections with a different color


I'm rendering a bunch of shapes on the screen (polygons), and I'm not using depth test. I simply want those shapes to use their own color when they're drawn over the empty space, and to to be red pixels whenever they're drawn over anything not empty, i.e. in my case over another polygon.

The problem here is really in the colors that I can work with and the factors that I can assign. So let's say that my color is something arbitrary, like RGBA(0.5, 0.7, 0.3, 1.0). This means as I draw first object, that's what's gonna end up in the Framebuffer.

then I draw the second object that overlaps, with the same color. But I want the overlap region to be red (1,0,0,1).

But here I run into problem, because no matter what sFactor and dFactor I assign for the blending function - I can never increase the values.

I obviously need to draw my original color with a factor of 1 whenever there's nothing in the Framebuffer, and 0 when there's something. So the first factor is and easy choice: GL_ONE_MINUS_DST_ALPHA.

But for the second factor, let's say I want to turn the original color RGBA(0.5, 0.7, 0.3, 1.0) into red RGBA(1, 0, 0, 1), and I obviously can't. Because my factor is clamped to [0,1], I can at most have the red channel of the result at 0.5, and not greater. I can negate the other 2 channels easily by providing GL_CONSTANT_COLOR as a factor, but I can't make any of the channels greater then the original.

Or can I? Is there any other way to achieve what I'm looking for?

Edit / Answer:

As was recommended, I used the stencil buffer, and it was exactly what I needed. I did not opt to make a separate framebuffer for it, but just work in the same, because it works better for me and promises less overhead. Anyway, here's the pseudocode:

        glEnable(GL_STENCIL_TEST);
        glStencilOp(GL_INCR,GL_INCR,GL_INCR);
        glStencilFunc(GL_ALWAYS,1,0xFF);
        glStencilMask(0xFF);
        glClear(GL_STENCIL_BUFFER_BIT);

        glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_ONE);
        renderer.render(stuff);

        // Parts of above stuff are erased / cleared
        glStencilOp(GL_ZERO,GL_ZERO,GL_ZERO);
        glBlendFunc(GL_ZERO,GL_ZERO);
        renderer.render(eraser);

        glStencilOp(GL_INCR,GL_INCR,GL_INCR);
        glBlendFunc(GL_ONE_MINUS_DST_ALPHA, GL_ONE);
        renderer.render(moreStuff)

        glStencilMask(0x00);                  //Disable writing to stencil buffer
        glBlendFunc(GL_ONE, GL_ZERO);         // render above everything
        glStencilFunc(GL_LESS,1,0xFF);        //only parts that are above 1 stencil value, i.e. intersections
                                              //this is a bit unintuitive
        renderer.render(overlay);             //quad that is filled with solid color, and takes the entire screen
        glDisable(GL_STENCIL_TEST);

it works great! I also included the part where I eraze some of the rendered image, just in case someone needs a similar task.

A few unintuitive points:

  1. glClear(GL_STENCIL_BUFFER_BIT) actually uses the glStencilMask(), so this has to be set to 0xFF for full clear. I made a mistake of putting the clear call above the mask assignment, so in essence it kept the mask from the previous render cycle 0x00 and wasn't clearing the stencil buffer. Which, if you want to play Leonardo with a paintbrush - is exactly what you want. ^_^
  2. glStencilFunc(func , ref, mast) actually works backwards from what is intuitive. the formula is: (ref & mask) func (stencil & mask). Ie glStencilFunc(GL_LESS,1,0xFF); will DISCARD any fragments that are greater than 1, because the formula is (1 & FF) < (stencil & FF), i.e. when stencil is 1 and 0 - the formula is FALSE, and the fragment is discarded. If the stencil is 2 and above, then 1<2 = true, and the fragment is kept.

Solution

  • Your problem is best solved with an aid of a stencil buffer.

    You render your primitives into an off-screen stencil buffer using glStencilOp(GL_INCR,GL_INCR,GL_INCR). This effectively counts how many primitives overlap every single pixel. When you're done, draw a full-screen quad with a shader that calculates the color based off the value in the stencil buffer (zero means a background color).