openglrenderingsdl-2alphablendingcompositing

How to get correct SourceOver alpha compositing in SDL with OpenGL

I am using an FBO (or "Render Texture") which has an alpha channel (32bpp ARGB) and clear that with a color that is not fully opaque, for example (R=1, G=0, B=0, A=0) (i.e. completely transparent). Then I am rendering a translucent object, for example a rectangle with color (R=1, G=1, B=1, A=0.5), on top of that. (All values normalized from 0 to 1)

According to common sense, as well as imaging software such as GIMP and Photoshop, as well as several articles on Porter-Duff compositing, I would expect to get a texture that is

Like so (you won't see this on the SO website):

Expected result, created with the gimp

Instead, the background color RGB values, which are (1.0, 0.0, 0.0) are weighted overall with (1 - SourceAlpha) instead of (DestAlpha * (1 - SourceAlpha)). The actual result is this:

Actual result

I have verified this behavior using OpenGL directly, using SDL's wrapper API, and using SFML's wrapper API. With SDL and SFML I have also saved the results as an image (with alpha channel) instead of merely rendering to the screen to be sure that it's not a problem with the final rendering step.

What do I need to do to produce the expected SourceOver result, either with SDL, SFML, or using OpenGL directly?

Some sources:

W3 article on compositing, specifies co = αs x Cs + αb x Cb x (1 – αs), weight of Cb should be 0 if αb is 0, no matter what.

English Wiki shows destination ("B") being weighted according to αb (as well as αs, indirectly).

German Wiki shows 50% transparency examples, clearly the transparent background's original RGB values do not interfere with either the green or the magenta source, also shows that the intersection is clearly asymmetric in favor of the element that is "on top".

There are also several questions on SO that seemingly deal with this at first glance, but I could not find anything that talks abut this specific issue. People suggest different OpenGL blending functions, but the general consensus seems to be glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA), which is what both SDL and SFML use by default. I have also tried different combinations with no success.

Another suggested thing is premultiplying the color with the destination alpha, since OpenGL can only have 1 factor, but it needs 2 factors for correct SourceOver. However, I cannot make sense of that at all. If I'm premultiplying (1, 0, 0) with the destination alpha value of, say, (0.1), I get (0.1, 0, 0) (as suggested here for example). Now I can tell OpenGL the factor GL_ONE_MINUS_SRC_ALPHA for this (and source with just GL_SRC_ALPHA), but then I'm effectively blending with black, which is incorrect. Though I am not a specialist on the topic, I put a fair amount of effort into trying to understand (and at least got to the point where I managed to program a working pure software implementation of every compositing mode). My understanding is that applying an alpha value of 0.1 "via premultiplication" to (1.0, 0.0, 0.0) is not at all the same as treating the alpha value correctly as the fourth color component.

Here is a minimal and complete example using SDL. Requires SDL2 itself to compile, optionally SDL2_image if you want to save as PNG.

// Define to save the result image as PNG (requires SDL2_image), undefine to instead display it in a window
#define SAVE_IMAGE_AS_PNG

#include <SDL.h>
#include <stdio.h>

#ifdef SAVE_IMAGE_AS_PNG
#include <SDL_image.h>
#endif

int main(int argc, char **argv)
{
    if (SDL_Init(SDL_INIT_VIDEO) != 0)
    {
        printf("init failed %s\n", SDL_GetError());
        return 1;
    }
#ifdef SAVE_IMAGE_AS_PNG
    if (IMG_Init(IMG_INIT_PNG) == 0)
    {
        printf("IMG init failed %s\n", IMG_GetError());
        return 1;
    }
#endif

    SDL_Window *window = SDL_CreateWindow("test", SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, 800, 600, SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN);
    if (window == NULL)
    {
        printf("window failed %s\n", SDL_GetError());
        return 1;
    }

    SDL_Renderer *renderer = SDL_CreateRenderer(window, 1, SDL_RENDERER_ACCELERATED | SDL_RENDERER_TARGETTEXTURE);
    if (renderer == NULL)
    {
        printf("renderer failed %s\n", SDL_GetError());
        return 1;
    }

    // This is the texture that we render on
    SDL_Texture *render_texture = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_RGBA8888, SDL_TEXTUREACCESS_TARGET, 300, 200);
    if (render_texture == NULL)
    {
        printf("rendertexture failed %s\n", SDL_GetError());
        return 1;
    }

    SDL_SetTextureBlendMode(render_texture, SDL_BLENDMODE_BLEND);
    SDL_SetRenderDrawBlendMode(renderer, SDL_BLENDMODE_BLEND);

    printf("init ok\n");

#ifdef SAVE_IMAGE_AS_PNG
    uint8_t *pixels = new uint8_t[300 * 200 * 4];
#endif

    while (1)
    {
        SDL_Event event;
        while (SDL_PollEvent(&event))
        {
            if (event.type == SDL_QUIT)
            {
                return 0;
            }
        }

        SDL_Rect rect;
        rect.x = 1;
        rect.y = 0;
        rect.w = 150;
        rect.h = 120;

        SDL_SetRenderTarget(renderer, render_texture);
        SDL_SetRenderDrawColor(renderer, 255, 0, 0, 0);
        SDL_RenderClear(renderer);
        SDL_SetRenderDrawColor(renderer, 255, 255, 255, 127);
        SDL_RenderFillRect(renderer, &rect);

#ifdef SAVE_IMAGE_AS_PNG
        SDL_RenderReadPixels(renderer, NULL, SDL_PIXELFORMAT_ARGB8888, pixels, 4 * 300);
        // Hopefully the masks are fine for your system. Might need to randomly change those ff parts around.
        SDL_Surface *tmp_surface = SDL_CreateRGBSurfaceFrom(pixels, 300, 200, 32, 4 * 300, 0xff0000, 0xff00, 0xff, 0xff000000);
        if (tmp_surface == NULL)
        {
            printf("surface error %s\n", SDL_GetError());
            return 1;
        }

        if (IMG_SavePNG(tmp_surface, "t:\\sdltest.png") != 0)
        {
            printf("save image error %s\n", IMG_GetError());
            return 1;
        }

        printf("image saved successfully\n");
        return 0;
#endif

        SDL_SetRenderTarget(renderer, NULL);
        SDL_SetRenderDrawColor(renderer, 255, 255, 255, 255);
        SDL_RenderClear(renderer);
        SDL_RenderCopy(renderer, render_texture, NULL, NULL);
        SDL_RenderPresent(renderer);
        SDL_Delay(10);
    }
}
Solution

  • Thanks to @HolyBlackCat and @Rabbid76 I was able to shed some light on this entire thing. I hope this can help out other people who want to know how about correct alpha blending and the details behind premultiplied alpha.

    The basic problem is that correct "Source Over" alpha blending in actually not possible with OpenGL's built-in blend functionality (that is glEnable(GL_BLEND), glBlendFunc[Separate](...), glBlendEquation[Separate](...)) (this is the same for D3D by the way). The reason is the following:

    When calculating the result color and alpha values of the blending operation (according to correct Source Over), one would have to use these functions:

    Each RGB color values (normalized from 0 to 1):

    RGB_f = ( alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s) ) / alpha_f

    The alpha value (normalized from 0 to 1):

    alpha_f = alpha_s + alpha_d x (1 - alpha_s)

    Where

    However, OpenGL can only handle a limited variety of additional factors to go with the source or destination values (RGB_s and RGB_d in the color equation) (see here), the relevant ones in this case being GL_ONE, GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA. We can specify the alpha formula correctly using those options, but the best we can do for RGB is:

    RGB_f = alpha_s x RGB_s + RGB_d x (1 - alpha_s)

    Which completely lacks the destination's alpha component (alpha_d). Note that this formula is equivalent to the correct one if \alpha_d = 1. In other words, when rendering onto a framebuffer which has no alpha channel (such as the window backbuffer), this is fine, otherwise it will produce incorrect results.

    To solve that problem and achieve correct alpha blending if alpha_d is NOT equal to 1, we need some gnarly workarounds. The original (first) formula above can be rewritten to

    alpha_f x RGB_f = alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s)

    if we accept the fact that the result color values will be too dark (they will be multiplied by the result alpha color). This gets rid of the division already. To get the correct RGB value, one would have to divide the result RGB value by the result alpha value, however, as it turns out that conversion usually never needed. We introduce a new symbol (pmaRGB) which denotes RGB values which are generally too dark because they have been multiplied by their corresponding pixel's alpha value.

    pmaRGB_f = alpha_s x RGB_s + alpha_d x RGB_d x (1 - alpha_s)

    We can also get rid of the problematic alpha_d factor by ensuring that ALL of the destination image's RGB values have been multiplied with their respective alpha values at some point. For example, if we wanted the background color (1.0, 0.5, 0, 0.3), we do not clear the framebuffer with that color, but with (0.3, 0.15, 0, 0.3) instead. In other words, we are doing one of the steps that the GPU would have to do already in advance, because the GPU can only handle one factor. If we are rendering to an existing texture, we have to ensure that it was created with premultiplied alpha. The result of our blending operations will always be textures that also have premultiplied alpha, so we can keep rendering things onto there and always be sure that the destination does have premultiplied alpha. If we are rendering to a semi-transparent texture, the semi-transparent pixels will always be too dark, depending on their alpha value (0 alpha meaning black, 1 alpha meaning the correct color). If we are rendering to a buffer which has no alpha channel (like the back buffer we use for actually displaying things), alpha_f is implicitly 1, so the premultiplied RGB values are equal to the correctly blended RGB values. This is the current formula:

    pmaRGB_f = alpha_s x RGB_s + pmaRGB_d x (1 - alpha_s)

    This function can be used when the source does not yet have premultiplied alpha (for example, if the source is a regular image that came out of an image processing program, with an alpha channel that is correctly blended with no premultiplied alpha).

    There is a reason we might want to get rid of \alpha_s as well, and use premultiplied alpha for the source as well:

    pmaRGB_f = pmaRGB_s + pmaRGB_d x (1 - alpha_s)

    This formula needs to be taken if the source happens to have premultiplied alpha - because then the source pixel values are all pmaRGB instead of RGB. This is always going to be the case if we are rending to an offscreen buffer with an alpha channel using the above method. It may also be reasonable to have all texture assets stored with premultiplied alpha by default so that this formula can always be taken.

    To recap, to calculate the alpha value, we always use this formula:

    alpha_f = alpha_s + alpha_d x (1 - alpha_s)

    , which corresponds to (GL_ONE, GL_ONE_MINUS_SRC_ALPHA). To calculate the RGB color values, if the source does not have premultiplied alpha applied to its RGB values, we use

    pmaRGB_f = alpha_s x RGB_s + pmaRGB_d x (1 - alpha_s)

    , which corresponds to (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA). If it does have premultiplied alpha applied to it, we use

    pmaRGB_f = pmaRGB_s + pmaRGB_d x (1 - alpha_s)

    , which corresponds to (GL_ONE, GL_ONE_MINUS_SRC_ALPHA).

    What that practically means in OpenGL: When rendering to a framebuffer with alpha channel, switch to the correct blending function accordingly and make sure that the FBO's texture always has premultiplied alpha applied to its RGB values. Note that the correct blending function may potentially be different for each rendered object, according to whether or not the source has premultiplied alpha. Example: We want a background [1, 0, 0, 0.1], and render an object with color [1, 1, 1, 0.5] onto it.

    // Clear with the premultiplied version of the real background color - the texture (which is always the destination in all blending operations) now complies with the "destination must always have premultiplied alpha" convention.
glClearColor(0.1f, 0.0f, 0.0f, 0.1f); 
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

//
// Option 1 - source either already has premultiplied alpha for whatever reason, or we can easily ensure that it has
//
{
    // Set the drawing color to the premultiplied version of the real drawing color.
    glColor4f(0.5f, 0.5f, 0.5f, 0.5f);

    // Set the blending equation according to "blending source with premultiplied alpha".
    glEnable(GL_BLEND);
    glBlendFuncSeparate(GL_ONE, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
    glBlendEquationSeparate(GL_ADD, GL_ADD);
}

//
// Option 2 - source does not have premultiplied alpha
// 
{
    // Set the drawing color to the original version of the real drawing color.
    glColor4f(1.0f, 1.0f, 1.0f, 0.5f);

    // Set the blending equation according to "blending source with premultiplied alpha".
    glEnable(GL_BLEND);
    glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
    glBlendEquationSeparate(GL_ADD, GL_ADD);
}

// --- draw the thing ---

glDisable(GL_BLEND);

    In either case, the resulting texture has premultiplied alpha. Here are 2 possibilities what we might want to do with this texture:

    If we want to export it as an image that is correctly alpha blended (as per the SourceOver definition), we need to get its RGBA data and explicitly divide each RGB value by the corresponding pixel's alpha value.

    If we want to render it onto the backbuffer (whose background color shall be (0, 0, 0.5)), we proceed as we would normally (for this example, we additionally want to modulate the texture with (0, 0, 1, 0.8)):

    // The back buffer has 100 % alpha.
glClearColor(0.0f, 0.0f, 0.5f, 1.0f); 
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

// The color with which the texture is drawn - the modulating color's RGB values also need premultiplied alpha
glColor4f(0.0f, 0.0f, 0.8f, 0.8f);

// Set the blending equation according to "blending source with premultiplied alpha".
glEnable(GL_BLEND);
glBlendFuncSeparate(GL_ONE, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glBlendEquationSeparate(GL_ADD, GL_ADD);

// --- draw the texture ---

glDisable(GL_BLEND);

    Technically, the result will have premultiplied alpha applied to it. However, because the result alpha will always be 1 for each pixel, the premultiplied RGB values are always equal to the correctly blended RGB values.

    To achieve the same in SFML:

    renderTexture.clear(sf::Color(25, 0, 0, 25));

sf::RectangleShape rect;
sf::RenderStates rs;
// Assuming the object has premultiplied alpha - or we can easily make sure that it has
{
    rs.blendMode = sf::BlendMode(sf::BlendMode::One, sf::BlendMode::OneMinusSrcAlpha);
    rect.setFillColor(sf::Color(127, 127, 127, 127));
}

// Assuming the object does not have premultiplied alpha
{
    rs.blendMode = sf::BlendAlpha; // This is a shortcut for the constructor with the correct blending parameters for this type
    rect.setFillColor(sf::Color(255, 255, 255, 127));
}

// --- align the rect ---

renderTexture.draw(rect, rs);

    And the likewise to draw the renderTexture onto the backbuffer

    // premultiplied modulation color
renderTexture_sprite.setColor(sf::Color(0, 0, 204, 204));
window.clear(sf::Color(0, 0, 127, 255));
sf::RenderStates rs;
rs.blendMode = sf::BlendMode(sf::BlendMode::One, sf::BlendMode::OneMinusSrcAlpha);
window.draw(renderTexture_sprite, rs);

    Unfortunately, this is not possible with SDL afaik (at least not on the GPU as part of the rendering process). Unlike SFML, which exposes fine-grained control over the blending mode to the user, SDL does not allow setting the individual blending function components - it only has SDL_BLENDMODE_BLEND hardcoded with glBlendFuncSeparate(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, GL_ONE, GL_ONE_MINUS_SRC_ALPHA).