[SOLVED] Monogame + HLSL Fisheye Vertex Shader

Monogame + HLSL Fisheye Vertex Shader

I'm working on a project for school where I need to implement a fisheye shader, and I'm struggling with the process for determining the depth values of the vertices. I know that for a point in space P1 = (x, y, z, w) that P * my projection matrix should result in a new point P2 = (a, b, c, d) where for any point between the near and far clipping planes P2.c/P2.d results in a number between 0 and 1.

My hlsl vertex shader contains this code where input.Position is the raw Vertex Position

float distance3d(float4 input) // Finds 3d hypotenuse
{
    return sqrt(input[0] * input[0] + input[1] * input[1] + input[3] * input[3]);
}
//Vertex Position data
    float4 worldPosition = mul(input.Position, World); // object to world xform
    float4 viewPosition = mul(worldPosition, View);  // world to camera xform
    output.Position = mul(viewPosition, Projection); // perspective xform
    float z = output.Position[3];
    float distance = distance3d(output.Position) * (z < 0 ? -1 : 1);
    float f = Projection[2][2];
    float n = Projection[3][2];
    output.Position[3] = distance;
    output.Position[2] = -distance * f + square * n;

And my projection matrix is generated using this code.

new Matrix(  -focalLength * Math.Sin(-FOV),                  0, 0, 0,
             0, -focalLength * Math.Sin(-FOV),                  0, 0,
             0, 0, -farPlane / (farPlane - nearPlane),            -1,
             0, 0, -farPlane / (farPlane - nearPlane) * nearPlane, 0);

Normal Projection: https://i.sstatic.net/nmS1R.jpg

Fisheye Projection: https://i.sstatic.net/KNwaz.jpg

As you can see, distant triangles render in front of near triangles.

Thanks for the help.

Solution

I solved my issue by changing from modifying the Projected z position (output.Position[3]) to modifying the z Position relative to the camera (viewPosition[2]). I output.Position[2] needs to include the hypotenuse instead of the raw z value for the depth calculations to work correctly.

    float4 worldPosition = mul(input.Position, World); // object to world xform
    float4 viewPosition = mul(worldPosition, View);  // world to camera xform
    float z = viewPosition[2];
    float square = distance3d(viewPosition) * (z < 0 ? -1 : 1);
    viewPosition[2] = square;
    output.Position = mul(viewPosition, Projection); // perspective xform
    output.wPosition = mul(input.Position, World);