glslshaderskybox

Calculate direction of pixel in fragment shader for use in procedural sky shader?


(So I'm in my barebones C++ Vulkan game engine and I want to make a procedural sky shader like there is in Godot 4 and so far it's been pretty straight forward, it's not a skybox but a single triangle large enough to cover the whole screen rendered orthographically in front of the camera but always rendered behind everything. In Godot 4, the procedural sky shader has a very important 3D vector called EYEDIR which is the direction of the current pixel in the fragment shader, I have not quite figured out how to calculate this.)

I tried the non-optimized code from this article to transform from window space to eye space https://www.khronos.org/opengl/wiki/Compute_eye_space_from_window_space My code looks like this:

vec4 calcEyeFromWindow()
{
    vec4 ndcPos;
    ndcPos.xy = (2.0 * gl_FragCoord.xy) / (vec2(screenWidth, screenHeight)) - 1;
    ndcPos.z = (2.0 * gl_FragCoord.z - near - far) / (far - near);
    ndcPos.w = 1.0;

    vec4 clipPos = ndcPos / gl_FragCoord.w;
    vec4 eyePos = invPers * clipPos;

    return eyePos;
}

Now I think I'm supposed to normalize that for a direction, however, it isn't what I'm looking for, it doesn't take into account the camera's rotation or field of view, it looks the same no matter where you turn and it just stretches to the window size no matter what you resize it to. eye space visualization in procedural sky shader

I've tried transforming the directions into the camera's perspective with

vec3 transformDirection(vec4 p){

    vec3 u1 = right * p.x;
    vec3 u2 = forward * p.y;
    vec3 u3 = up * p.z;
    return normalize(u1 + u2 + u3);// the transformed direction in world space
}

but not only does this still not account for the field of view, it doesn't even work anyway, the sky appears to twist and flip around as you look around with the camera. (yes, Z is up)

So basically I'm asking, how is EYEDIR calculated in Godot 4 sky shaders? (What is the math for figuring out the direction of each pixel in a fragment shader which assumedly takes FOV into account and aligns perfectly with the other things being rendered in the scene?)


Solution

  • So after I've recently began rewriting the engine (I guess this is 4 months later, feels like a year later lol), I have found some optimizations and I think I just understand matrix math a bit better now than I did. Also, I use Microsoft Copilot for a lot of my stupid questions now and it is very helpful and only sometimes stupider than I am, this is one of those situations.

    My sky shader is just a single triangle mesh of vec2(0), vec2(0,2), and vec2(2,0). It's also worth mentioning I switched to using Y as the up axis which has helped tremendously because every time I tried to yoink code from the internet or ask an AI for help it assumed Y was up and I'd have to hack it into being a Z up. Y up is just better and makes more sense for a game engine. The sky mesh has no Z, so I don't need the weird "near - far" bs in the calcEyeFromWindow function. But most importantly, since my verts roughly line up with the screen coords after I do this: vec2 v = inPosition * vec2(ubo.screenWidth, ubo.screenHeight); in the vert shader, it means I can also do calcEyeFromWindow in the vert shader rather than the frag shader. Unfortunately I still need to normalize EYEDIR in the frag shader, which is the most expensive part, but it still saves a lot of calculation because I had to normalize it anyway. So in case anyone is doing the same thing, here is a basic version of the updated vert shader:

    #version 450
    
    layout( binding = 0) uniform UniformBufferObject {
        float screenWidth;
        float screenHeight;
        float dayIntensity; // 1: high noon, -1: midnight
        mat4 proj;
        mat4 invPers;
        mat4 cameraTransform;
        mat4 starmapTransform;
    } ubo;
    
    layout(location = 0) in vec2 inPosition;
    
    layout(location = 1) out float dayIntensity;
    layout(location = 2) out mat3 starmapTransform;
    layout(location = 5) out vec3 eyeDir;
    
    const mat4 viewMatrix = mat4(vec4(1,0,0,0), vec4(0,1,0,0),vec4(0,0,0,0),vec4(0,0,0,1));
    
    vec3 calcEyeFromWindow(vec2 v)
    {
        vec4 ndcPos = vec4(0, 0, 0, 1);
        ndcPos.xy = (2.0 * v) / (vec2(ubo.screenWidth, ubo.screenHeight)) - 1;
        vec4 eyePos = ubo.invPers * ndcPos;
        return eyePos.xyz;
    }
    
    void main() {
    
        vec2 v = inPosition * vec2(ubo.screenWidth, ubo.screenHeight);
        gl_Position = ubo.proj * viewMatrix * vec4(v, 0.0, 1.0);
    
        dayIntensity = ubo.dayIntensity;
        starmapTransform = mat3(ubo.starmapTransform);
    
        eyeDir = mat3(ubo.cameraTransform) * calcEyeFromWindow(v);
    }
    

    and in the frag shader I need to only do this vec3 EYEDIR = normalize(eyeDir); And btw, in this particular case the inverse projection matrix does not need its Y flipped like it normally does in Vulkan.

    my sky shader showing the starmap and the Vulkan tutorial Viking room flipped on its side from me switching to Y up instead of Z up