openglglsl

How to ensure screen space derivatives are present on triangle edges?


I draw an anti-aliased rounded square as ground plane for models to sit on top using the following vertex shader:

#version 330
out vec2 tex;
uniform mat4 mvp;

const vec2 pos[4] = vec2[](
    vec2(-1.0, 1.0),
    vec2(1.0, 1.0),
    vec2(1.0, -1.0),
    vec2(-1.0, -1.0));

void main()
{
    tex = pos[gl_VertexID];
    /* setup camera */
    gl_Position = mvp * vec4(pos[gl_VertexID], 0.0, 1.0);
}

And the following fragment shader:

#version 330
in vec2 tex;
out vec4 Out_Color;

float roundedBoxSDF(vec2 uv, float Size, float Radius)
{
    return length(max(abs(uv) - Size + Radius, 0.0)) - Radius;
}

void main()
{
    /* 95% to make sure there is a screen space derivative present to
    calculate against. */
    float dist = roundedBoxSDF(tex, 0.95, 0.5);
    float smoothedAlpha = dist / length(vec2(dFdx(dist), dFdy(dist)));

    Out_Color = vec4(vec3(1, 1, 1), 1.0 - smoothedAlpha);
}

If I draw said square with texture coordinates right up to the edge of the quad, the non-round edges along the quad's side are always aliased, because the signed distance field has no screen space derivative to calculate against the edge with distance 1 away from the origin.

enter image description here

Seems like an easy fix: As already in the fragment shader above, I simply shrink the texture coordinates to 95% roundedBoxSDF(tex, 0.95, 0.5);, so there is a quad without edge to gather the screen space derivative against. Nicely anti-aliased...

enter image description here

...until I zoom far out out or view at an oblique angle. Then the 5% margin is not enough anymore and I get aliasing again, because screen-space wise there is no pixel present to get said derivative.

enter image description here

How can I account for something like this? Continue shrinking the texture coordinates and growing the vertices as a function of distance from the camera? Is there anything smarter than that?


Solution

  • The problem is not with derivatives — contrary to what you believe, they are well-defined on the edges. Nor is the lack of anisotropic filtering — because it happens at non-oblique angles too.

    As you said yourself, the problem is that shrinking by 0.95 is not enough at some angles. That is because the analytical edge of your shape gets closer to the quad edge by less than a pixel. In such a case there simply isn't any pixel rasterized where a non-transparent pixel needs to be:

    enter image description here

    If you go the shrinking route, the amount you shrink itself depends on the derivatives — you need to shrink by a pixel in screen-space. An equivalent but simpler method would be to calculate the SDF on the interior of your non-shrunk box, then set out_Color.a = -smoothedAlpha:

    enter image description here

    In either case shrinking is a rather hackish way of fixing the problem, because it changes the effective size of your shape.

    The more correct way would be to dilate the rasterized triangle by a pixel. There is an NVidia extension that does exactly that: NV_conservative_raster_dilate. In unextended OpenGL you can get similar results with a geometry shader. In either case the dilated triangles will operlap, which requires extra measures if your shape is going to be partially transparent.