First, some pictures. Here is a CORRECT render done on a Quadro P2200, which is the main dev machine because of high compatibilty with mobile graphics in speed an acceleration:
This is the image in wireframe, this a specific test put together to figure out the z-problem
This is the image in wireframe, this a specific test put together to figure out the z-problem.
Now here is the exact same setup rendered on an NVidia GeForce RTX 4060:
So as it gets far away, the z starts bleeding through. This is a 24-bit zbuffer, BTW.
Here is how the perspective matrix is created:
float aAspect = (float) theRez.mX / (float) theRez.mY;
float aNear = 1;
float aFar = 100
float aWidth = gMath.Cos(theFOV / 2.0f);
float aHeight = gMath.Cos(theFOV / 2.0f);
if (aAspect > 1.0) aWidth /= aAspect;
else aHeight *= aAspect;
float s = gMath.Sin(theFOV / 2.0f);
float aDist = 1.0f - aNear / aFar;
Matrix aPerspectiveMatrix;
aPerspectiveMatrix.mData.m[0][0] = aWidth;
aPerspectiveMatrix.mData.m[1][0] = 0;
aPerspectiveMatrix.mData.m[2][0] = gG.m3DOffset.mX/theRez.mX/2;
aPerspectiveMatrix.mData.m[3][0] = 0;
aPerspectiveMatrix.mData.m[0][1] = 0;
aPerspectiveMatrix.mData.m[1][1] = aHeight;
aPerspectiveMatrix.mData.m[2][1] = gG.m3DOffset.mY/theRez.mY/2;
aPerspectiveMatrix.mData.m[3][1] = 0;
aPerspectiveMatrix.mData.m[0][2] = 0;
aPerspectiveMatrix.mData.m[1][2] = 0;
aPerspectiveMatrix.mData.m[2][2] = s / aDist;
aPerspectiveMatrix.mData.m[3][2] = -(s * aNear / aDist);
aPerspectiveMatrix.mData.m[0][3] = 0;
aPerspectiveMatrix.mData.m[1][3] = 0;
aPerspectiveMatrix.mData.m[2][3] = s;
aPerspectiveMatrix.mData.m[3][3] = 0;
So issue seems to be that matrix (2,2) and (3,2) are odd numbers-- in this test, (2,2) contains 26.4104 and (3,2) contains -.264101 ... if I change these numbers to be -.01 and -1, the z-fighting goes away (but so does the distance clipping).
If I go into the shader and do this:
zdepth=1.0f-smoothstep(-.264101,.264101,(pixel.zdepth));
it also fixes it (though I have to tweak some render settings because it's a reverse z-buffer). I don't want to add the extra work to the shader if it's possible to fix it some other way. Nor do I want to do the regression testing for all the shaders that will need this!
The issue here seems to be that although these numbers work fine on the Quadro, on a more modern card they're not. I THINK it might be some OpenGL default that is set up in the Quadro driver but not the NVidia one, but I haven't been able to find an inconsistency yet by peeking at things with glGet.
The perspective matrix is set up in an unorthodox way, but I'm dealing with legacy code to modernize an old project here. Attempting to plug in a more orthodox perspective setup leaves me rendering a blank screen, so I have to figure this one out.
Can anyone suggest something I might look at or try to get these two different cards to render the same thing?
-------------- (Edit to add more information) -----------------
I did a manual transform against the perspective matrix of a near and far vertex. The nearest red square is centered on 0,0,0 and the farthest red square is centered on 0,-50,0 to give an idea of the range and the scale of the scene. The camera is 15 units away from "lookat" (which in this case is the center of the foremost red quad).
When run through perspective, the near one evaluates z as ~.95 and the far one evaluates z as ~1.0 ... so I can see that the problem is simply "my whole scene is crammed into .95-1.0 and literally 95% of my z-buffer is being used by the blank space between the camera and the nearest square."
Why does this work fine on the Quadro card?????? Does it have a completely different z-equation??? Does it have an OpenGL default that my more modern card defaults to something different?
The issue remains, how to fix this. Right now my near/far are set as 1,50 ... if I change them to 5,50 I get improvement, but it's still rendering the whole scene within .75-1.0 and I still get artifacts, just less. And I can't set it that high anyway because things need to get close to the camera sometimes.
Ideally, everything visible in this scene would occupy, say, .25-1.0 of the z-buffer, or even .15-1.0.
Is there a way to accomplish this without taking over the z-computation completely in the shader to make it more linear?
Posting an answer to this, because maybe it will help some people who are porting legacy code. Just a reminder of things to look at, if you're seeing discrepancies between older and newer cards.
In all the shaders of this project was the line
precision mediump float;
...thereby turning all floats into 2-byte low-resolution versions of themselves. The older card either just ignores precision or improperly implements it, so all the floats remained 4-byte.
I was so focused on the zbuffer that it didn't even occur to me to look into shader code. And even if I had, I would have assumed that lowp was two bytes, mediump was four bytes, and highp was 8 bytes, on floats. It may be possible that my older card interprets them this way.
By changing this line in all the shaders, the visuals on the new card now matches the visuals on the old card.