Normal mapping on flat water surface produces incorrect specular highlights

I have a flat water surface with a dudv and a normal map attached to it. The dudv map works correct and the normal map is attached correct as well (visualizing the normal map looks like it should). The specular highlights are always showing at the wrong place though as if the light direction is incorrect. The lighting works correct without normal mapping, so I don't believe it is the light direction, but probably something with the tangent space. Since I calculate the tangent space from a static set of vectors I'm confused where it could go wrong.

Here in the vertex shader I create the TBN matrix that I use to create the tangent space vectors that I send to the fragment shader:

const vec3 TANGENT = vec3(1.0, 0.0, 0.0);
const vec3 NORMAL = vec3(0.0, 1.0, 0.0);
const vec3 BITANGENT = vec3(0.0, 0.0, -1.0);

out vec3 FragPos;
out vec3 TangentFragPos;
out vec3 TangentLightDir;
out vec3 TangentPlayerPos;

void main()
{
    FragPos = vec3(model * vec4(vertex, 1.0));
    mat3 mod = transpose(inverse(mat3(model)));
    [...]
    vec3 n = normalize(mod * NORMAL);
    vec3 t = normalize(mod * TANGENT);
    vec3 b = normalize(mod * BITANGENT);
    mat3 TBN = transpose(mat3(t, b, n));
    TangentFragPos =  TBN * FragPos;
    TangentLightDir =  TBN * sun.Position.xyz;
    TangentPlayerPos =  TBN * playerPos;
}

In the fragment shader I then sample a normal vector from the normal map and use the transformed tangent space vectors to calculate specular highlights:

in vec3 FragPos;
in vec3 TangentFragPos;
in vec3 TangentLightDir;
in vec3 TangentPlayerPos;

uniform sampler2D normalMap;

void main()
{    
    [...]
    vec3 normal = texture(normalMap, vec2(TexCoords * TextureScale) + vec2(Time)).xyz;
    normal = normalize(normal * 2.0 - 1.0);
    // normal = vec3(0.0, 0.0, 1.0); // this gives proper specular highlights, but not mapped

    // Specular lighting
    vec3 lightDir = normalize(-TangentLightDir);
    viewDir = normalize(TangentPlayerPos - TangentFragPos);
    vec3 reflectDir = normalize(reflect(-lightDir, normal));
    float spec = pow(max(dot(viewDir, reflectDir), 0.0), 64);
    vec3 lighting = sun.Specular * spec * fresnel;

    color.xyz *= lighting;       
}

Note that I do the lighting in world-space.

The lighting works fine without normal mapping, but as soon as I introduce the TBN matrix into the equations the specular highlights are not in the right direction on the water.

EDIT

I've added an image to see how it currently looks and where the specular light should be. That might add some extra insight into the problem:

 

EDIT2

Here's the weird thing. If I manually define the normal vector as vec3(0.0, 0.0, 1.0) (pointing upwards in tangent space) I get perfect specular highlights (but with no mapped variation) and once I take the normal from the normal map, I get incorrect highlights again so I'd say the cause of the issue would be at the normals. The normal map I'm using however is a default normal map you'd usually see as you can see below:

Why is it that once I take a normal from this normal map (which already should be in tangent space), the specular highlights break down?

It's been more than a week of debugging now and I finally found the issue. I've been using gamma correction and my texture class by defaults loads texture with the GL_SRGB property as the texture's internal format so OpenGL properly converts gamma corrected textures to their linear brothers.

The problem was that I also loaded the dudv and Normal map with this texture class without changing this property so the normal and dudv maps had OpenGL's gamma->linear correction applied and thus gave incorrect results.

Loading these textures with GL_RGB solved the issue.