LibGDX texture blending with OpenGL blending function

In libGdx, i'm trying to create a shaped texture: Take a fully-visible rectangle texture and mask it to obtain a shaped textured, as shown here:

Here I test it on rectangle, but i will want to use it on any shape. I have looked into this tutorial and came with an idea to first draw the texture, and then the mask with blanding function:

batch.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);

• GL20.GL_ZERO - because i really don't want to paint any pixels from the mask
• GL20.GL_SRC_ALPHA - from original texture i want to paint only those pixels, where mask was visible (= white).

Crucial part of the test code:

batch0.enableBlending();
batch0.begin();

batch0.draw(original, 0, 0); //to see the original
batch0.draw(mask, width1, 0); //and the mask

batch0.draw(original, 0, height1); //base for the result

batch0.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);
batch0.draw(mask, 0, height1); //draw mask on result
batch0.setBlendFunction(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);

batch0.end();

The center ot the texture get's selected well, but instead of transparent color around, i see black:

Why is the result blank and not transparent?

(Full code - Warning: very messy)

What you're trying to do looks like a pretty clever use of blending. But I believe the exact way you apply it is "broken by design". Let's walk through the steps:

1. You render your background with red and green squares.
2. You render an opaque texture on top of you background.
3. You erase parts of the texture you rendered in step 2 by applying a mask.

The problem is that for the parts you erase in step 3, the previous background is not coming back. It really can't, because you wiped it out in step 2. The background of the whole texture area was replaced in step 2, and once it's gone there's no way to bring it back.

Now the question is of course how you can fix this. There are two conventional approaches I can think of:

• You can combine the texture and mask by rendering them into an off-sreen framebuffer object (FBO). You perform steps 1 and 2 as you do now, but render into an FBO with a texture attachment. The texture you rendered into is then a texture with alpha values that reflect your mask, and you can use this texture to render into your default framebuffer with standard blending.
• You can use a stencil buffer. Masking out parts of rendering is a primary application of stencil buffers, and using stencil would definitely be a very good solution for your use case. I won't elaborate on the details of how exactly to apply stencil buffers to your case in this answer. You should be able to find plenty of examples both online and in books, including in other answers on this site, if you search for "OpenGL stencil". For example this recent question deals with doing something similar using a stencil buffer: OpenGL stencil (Clip Entity).

So those would be the standard solutions. But inspired by the idea in your attempt, I think it's actually possible to get this to work with just blending. The approach that I came up with uses a slightly different sequence and different blend functions. I haven't tried this out, but I think it should work:

1. You render the background as before.
2. Render the mask. To prevent it from wiping out the background, disable writing to the color components of the framebuffer, and only write to the alpha component. This leaves the mask in the alpha component of the framebuffer.
3. Render the texture, using the alpha component from the framebuffer (DST_ALPHA) for blending.

You will need a framebuffer with an alpha component for this to work. Make sure that you request alpha bits for your framebuffer when setting up your context/surface.

The code sequence would look like this:

// Draw background.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glDisable(GL_BLEND);
// Draw mask.
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glEnable(GL_BLEND);
glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA);
// Draw texture.