[SOLVED] How to properly transform normals for diffuse lighting in OpenGL?

How to properly transform normals for diffuse lighting in OpenGL?

In the attempt to get diffuse lighting correct, I read several articles and tried to apply them as close as possible.

However, even if the transform of normal vectors seems close to be right, the lighting still slides slightly over the object (which should not be the case for a fixed light).

Note 1: I added bands based on the dot product to make the problem more apparent.
Note 2: This is not Sauron eye.

In the image two problems are apparent:

The normal is affected by the projection matrix: when the viewport is horizontal, the normals display an elliptic shading (as in the image). When the viewport is vertical (height>width), the ellipse is vertical.
The shading move over the surface when the camera is rotated around the object.This is not much visible with normal lighting, but get apparent when projecting patterns from the light source.

Code and attempts:

Unfortunately, a minimal working example get soon very large, so I will only post relevant code. If this is not enough, as me and I will try to publish somewhere the code.

In the drawing function, I have the following matrix creation:

glm::mat4 projection = glm::perspective(45.0f, (float)m_width/(float)m_height, 0.1f, 200.0f);
glm::mat4 view = glm::translate(glm::mat4(1), glm::vec3(0.0f, 0.0f, -2.5f))*rotationMatrix; // make the camera 2.5f away, and rotationMatrix is driven by the mouse.
glm::mat4 model = glm::mat4(1); //The sphere at the center.
glm::mat4 mvp = projection * view * model;
glm::mat4 normalVp = projection * glm::transpose(glm::inverse(view * model));

In the vertex shader, the mvp is used to transform position and normals:

#version 420 core

uniform mat4 mvp;
uniform mat4 normalMvp;

in vec3 in_Position;
in vec3 in_Normal;
in vec2 in_Texture;

out Vertex
{
    vec4 pos;
    vec4 normal;
    vec2 texture;
} v;

void main(void)
{
    v.pos = mvp * vec4(in_Position, 1.0);
    gl_Position = v.pos;
    v.normal = normalMvp * vec4(in_Normal, 0.0);
    v.texture = in_Texture;
}

And in the fragment shader, the diffuse shading is applied:

#version 420 core

in Vertex
{
    vec4 pos;
    vec4 normal;
    vec2 texture;
} v;

uniform sampler2D uSampler1;
out vec4 out_Color;

uniform mat4 mvp;
uniform mat4 normalMvp;
uniform vec3 lightsPos;
uniform float lightsIntensity;

void main()
{

    vec3 color = texture2D(uSampler1, v.texture);
    vec3 lightPos = (mvp * vec4(lightsPos, 1.0)).xyz;
    
    vec3 lightDirection = normalize( lightPos - v.pos.xyz );
    float dot = clamp(dot(lightDirection, normalize(v.normal.xyz)), 0.0, 1.0);

    vec3 ambient = 0.3 * color;
    vec3 diffuse = dot * lightsIntensity * color;

    // Here I have my debug code to add the projected bands on the image.
    //     kind of if(dot>=0.5 && dot<0.75) diffuse +=0.2;...

    vec3 totalLight = ambient + diffuse;

    out_Color = vec4(totalLight, 1.0);
}

Question:

How to properly transform the normals to get diffuse shading?

Mostly, all sources agree that it should be enough to multiply the projection matrix by the transpose of the inverse of the model-view matrix. That is what I think I am doing, but the result is not right apparently.

Solution

Lighting calculations should not be performed in clip space (including the projection matrix). Leave the projection away from all variables, including light positions etc., and you should be good.

Why is that? Well, lighting is a physical phenomenon that essentially depends on angles and distances. Therefore, to calculate it, you should choose a space that preserves these things. World space or camera space are two examples of angle and distance-preserving spaces (compared to the physical space). You may of course define them differently, but in most cases they are. Clip space preserves neither of the two, hence the angles and distances you calculate in this space are not the physical ones you need to determine physical lighting.