How do you fix the pitch/lookAt flip when you rotate pitch past 90 degrees?

I've been following the LearnOpenGL tutorial series and I got to the section on camera, and while they've set up a fly camera, I've been trying to adapt it to more of an FPS-style camera. It usually works fine, but whenever I try to look directly up or down, the pitch seems to flip, up is down and down is up.

void Camera::matrix(float fov, float nearPlane, float farPlane, Shader &shader, const char* uniform) {

  mat4 view = mat4(1.0f);
  mat4 proj = mat4(1.0f);

  direction.x = cos(glm::radians(yaw)) * cos(glm::radians(pitch));
  direction.y = sin(glm::radians(pitch));
  direction.z = sin(glm::radians(yaw)) * cos(glm::radians(pitch));
  front = normalize(direction);

  view = lookAt(pos, pos + front, up);
  proj = perspective(radians(fov), ((float)width / (float)height), nearPlane, farPlane);

  glUniformMatrix4fv(glGetUniformLocation(shader.shaderProgram, uniform), 1, GL_FALSE, glm::value_ptr(proj * view));

}

void Camera::look() {

  newMouseX = getMouseX();
  newMouseY = getMouseY();

  float dx = (float) (newMouseX - oldMouseX) * getDeltaTime();
  float dy = (float) (newMouseY - oldMouseY) * getDeltaTime();

  yaw += dx * sensitivity;
  pitch += dy * sensitivity;

  oldMouseX = newMouseX;
  oldMouseY = newMouseY;

}

I tried clamping the rotation so that it simply can't rotate past 90 degrees pitch, and while it works and feels fairly natural, it just feels like a quick solution that could cause problems later down the line. I think it would be a more natural solution to allow the camera to rotate past 90 degrees, but I just don't really know how I'd go about doing that.

Welcome to the long standing problem of how to represent and accumulate orientations.

The representation you are using, separate pitch and yaw, is more formally known as Euler angles. And as you've discovered, weird stuff happens when you flip around the axes. There are a few different solutions:

Stick to Euler angles, but have a fixed sequence of tests every time you update the values. "if the new pitch is > 90 or < -90, adjust the yaw ..."

Every update, convert the existing angles and the change to pitch/bank into matrices, multiply them, convert back to angles.

Instead of using Euler angles, just store orientation as a matrix throughout. Or learn about quaternions.

There's no perfect solution. For more detail any book about 3D game programming, and most about 3D graphics in general, should have a section on how to manage orientation. I quite like "3D Math Primer for Graphics and Game Development" by Dunn and Parberry.