Initial situation

I want to visualize simulation data in openGL. My data consists of particle positions (x, y, z) where each particle has some properties (like density, temperature, ...) which will be used for coloring. Those (SPH) particles (100k to several millions), grouped together, actually represent planets, in case you wonder. I want to render those particles as small 3D spheres and add ambient, diffuse and specular lighting.

Status quo and questions

1. In MY case: In which coordinate frame do I do the lightning calculations? Which way is the "best" to pass the various components through the pipeline?

I saw that it is common to do it in view space which is also very intuitive. However: The normals at the different fragment positions are calculated in the fragment shader in clip space coordinates (see appended fragment shader). Can I actually convert them "back" into view space to do the lightning calculations in view space for all the fragments? Would there be any advantage compared to doing it in clip space?

2. It would be easier to get the normals in view space if I would use meshes for each sphere but I think with several million particles this would decrease performance drastically, so better do it with sphere intersection, would you agree?

PS: I don't need a model matrix since all the particles are already in place.

//VERTEX SHADER

#version 330 core
layout (location = 0) in vec3 position;
layout (location = 2) in float density;

uniform float radius;
uniform vec3 lightPos;
uniform vec3 viewPos;

out vec4 lightDir;
out vec4 viewDir;
out vec4 viewPosition;
out vec4 posClip;
out float vertexColor;


// transformation matrices
uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;

void main()
{
    lightDir        = projection * view * vec4(lightPos - position, 1.0f);
    viewDir         = projection * view * vec4(viewPos - position, 1.0f);
    viewPosition    = projection * view * vec4(lightPos, 1.0f);
    posClip         = projection * view * vec4(position, 1.0f);

    gl_Position = posClip;
    gl_PointSize = radius;

    vertexColor = density;

}

3. I know that projective divion happens for the gl_Position variable, does that actually happen to ALL vec4's which are passed from the vertex to the fragment shader? If not, maybe the calculations in the fragment shader would be wrong?

And the fragment shader where the normals and diffuse/specular lightning calculations in clip space:

//FRAGMENT SHADER

#version 330 core

in float vertexColor;
in vec4 lightDir;
in vec4 viewDir;
in vec4 posClip;
in vec4 viewPosition;

uniform vec3 lightColor;

vec4 colormap(float x); // returns vec4(r, g, b, a)

out vec4 vFragColor;


void main(void)
{
    // AMBIENT LIGHT
    float ambientStrength = 0.0;
    vec3 ambient = ambientStrength * lightColor;

    // Normal calculation done in clip space (first from texture (gl_PointCoord 0 to 1) coord to NDC( -1 to 1))
    vec3 normal;
    normal.xy = gl_PointCoord * 2.0 - vec2(1.0);    // transform from 0->1 point primitive coords to NDC -1->1
    float mag = dot(normal.xy, normal.xy);          // sqrt(x=1) = sqrt(x)
    if (mag > 1.0)                                  // discard fragments outside sphere
        discard;            
    normal.z = sqrt(1.0 - mag);                     // because x^2 + y^2 + z^2 = 1

    // DIFFUSE LIGHT
    float diff = max(0.0, dot(vec3(lightDir), normal));
    vec3 diffuse = diff * lightColor;

    // SPECULAR LIGHT
    float specularStrength = 0.1;
    vec3 viewDir = normalize(vec3(viewPosition) - vec3(posClip));
    vec3 reflectDir = reflect(-vec3(lightDir), normal);  
    float shininess = 64;
    float spec = pow(max(dot(vec3(viewDir), vec3(reflectDir)), 0.0), shininess);
    vec3 specular = specularStrength * spec * lightColor;  

     vFragColor = colormap(vertexColor / 8) * vec4(ambient + diffuse + specular, 1);

} 

4. Now this actually "kind of" works but i have the feeling that also the sides of the sphere which do NOT face the light source are being illuminated, which shouldn't happen. How can I fix this?

Some weird effect: In this moment the light source is actually BEHIND the left planet (it just peaks out a little bit at the top left), bit still there are diffuse and specular effects going on. This side should be actually pretty dark! =(

Also at this moment I get some glError: 1282 error in the fragment shader and I don't know where it comes from since the shader program actually compiles and runs, any suggestions? :)

The things that you are drawing aren't actually spheres. They just look like them from afar. This is absolutely ok if you are fine with that. If you need geometrically correct spheres (with correct sizes and with a correct projection), you need to do proper raycasting. This seems to be a comprehensive guide on this topic.

1. What coordinate system?

In the end, it is up to you. The coordinate system just needs to fulfill some requirements. It must be angle-preserving (because lighting is all about angles). And if you need distance-based attenuation, it should also be distance-preserving. The world and the view coordinate systems usually fulfill these requirements. Clip space is not suited for lighting calculations as neither angles nor distances are preserved. Furthermore, gl_PointCoord is in none of the usual coordinate systems. It is its own coordinate system and you should only use it together with other coordinate systems if you know their relation.

2. Meshes or what?

Meshes are absolutely not suited to render spheres. As mentioned above, raycasting or some screen-space approximation are better choices. Here is an example shader that I used in my projects:

#version 330

out vec4 result;

in fData
{
    vec4 toPixel; //fragment coordinate in particle coordinates
    vec4 cam;     //camera position in particle coordinates
    vec4 color;   //sphere color
    float radius; //sphere radius
} frag;

uniform mat4 p; //projection matrix

void main(void)
{
    vec3 v = frag.toPixel.xyz - frag.cam.xyz;
    vec3 e = frag.cam.xyz;
    float ev = dot(e, v);
    float vv = dot(v, v);
    float ee = dot(e, e);
    float rr = frag.radius * frag.radius;

    float radicand = ev * ev - vv * (ee - rr);
    if(radicand < 0)
        discard;

    float rt = sqrt(radicand);


    float lambda = max(0, (-ev - rt) / vv); //first intersection on the ray
    float lambda2 = (-ev + rt) / vv;  //second intersection on the ray
    if(lambda2 < lambda) //if the first intersection is behind the camera
        discard;

    vec3 hit = lambda * v; //intersection point
    vec3 normal = (frag.cam.xyz + hit) / frag.radius;

    vec4 proj = p * vec4(hit, 1); //intersection point in clip space
    gl_FragDepth = ((gl_DepthRange.diff * proj.z / proj.w) + gl_DepthRange.near + gl_DepthRange.far) / 2.0;

    vec3 vNormalized = -normalize(v);
    float nDotL = dot(vNormalized, normal);
    vec3 c = frag.color.rgb * nDotL + vec3(0.5, 0.5, 0.5) * pow(nDotL, 120);    

    result = vec4(c, frag.color.a);
}

3. Perspective division

Perspective division is not applied to your attributes. The GPU does perspective division on the data that you pass via gl_Position on the way to transforming them to screen space. But you will never actually see this perspective-divided position unless you do it yourself.

4. Light in the dark

This might be the result of you mixing different coordinate systems or doing lighting calculations in clip space. Btw, the specular part is usually not multiplied by the material color. This is light that gets reflected directly at the surface. It does not penetrate the surface (which would absorb some colors depending on the material). That's why those highlights are usually white (or whatever light color you have), even on black objects.