OpenGL application rotate Camera

I'm trying to implement a simple application in which the user can look around itself by changing the parameters of gluLookAt function. I've implemented the rotation around y axis in the following way:

float cx1;
float cz1;

float d1 = cx - eyex;
float d2 = cz - eyez;

cx1 = d2*sin(alfa*PI / 180) + d1*cos(alfa*PI / 180);
cz1 = d2*cos(alfa*PI / 180) - d1*sin(alfa*PI / 180);
yRotationAngle = yRotationAngle +alfa;

cx = cx1 + eyex;
cz = cz1 + eyez;

This works fine, but when I'm trying to rotate around x axis I don't get the desired behavior. I wanted to rotate the camera "up" and "down" but in that specific direction in which I am in that moment. The code for rotating around x axis is implemented in the following way: float d1 = cy - eyey; float d2 = cz - eyez;

//rotate around x
cy1 = cos(angleToRotate) * d1 - sin(angleToRotate) * d2;
cz1 = sin(angleToRotate) * d1 + cos(angleToRotate) * d2;

cy = cy1 + eyey;
cz = cz1 + eyez;

Do I need necessarily some inverse rotations around y axis before I rotate around x? If so, what would be the value of the angle needed to rotate?

Solution

The math for the second rotation would be slightly more complicated because you want to apply it relative to the current position. But I think you can make this whole thing much simpler if you keep track of the current angles, and add/subtract from them when rotating the camera. Essentially, you primarily operate in spherical coordinate space, and only calculate cartesian coordinates when it's time to apply the rotation.

To do this, you need two angles. I'll use the following nomenclature:

yawAng: Current rotation angle around the y-axis, measured relative to the z-axis.
pitchAng: Angle relative to the xz-plane. Positive for up (in direction of positive y, negative for down).

With this, the basic camera rotations simply become increments/decrements to these angles. For an increment of angInc:

Left: yawAng += angInc
Right: yawAng -= angInc
Up: pitchAng += angInc
Down: pitchAng -= angInc

Then, when it's time to apply the gluLookAt() transformation, you convert the spherical coordinates to cartesian coordinates using the standard formula. With distance d between camera and center:

cx = eyex + d * sin(yawAng) * cos(pitchAng);
cy = eyey + d * sin(pitchAng)
cz = eyez + d * cos(yawAng) * cos(pitchAng);