[SOLVED] Vulkan culling problem after flipping the MVP upside down

Vulkan culling problem after flipping the MVP upside down

I ran a vulkan renderer on an android device, the resulted image is upside down. I don't want to put a negative height in the VkViewport definition, because there is code and comments that suggest that the matrix is inverted into Y.

I tried to invert the sign of mvp[5] matrix array in the following code, but got culling problem in the rendered image.

// update opengl projection matrix to vulkan conventions:
// Vulkan clip space has inverted Y and half Z  ( Z - [0, 1] instead of [-1, 1])
float zNear = r_znear->value;
float m10 = -zFar / (zFar - zNear);

float py = tan(fovY * (M_PI / 360.0f));
float px = tan(fovX * (M_PI / 360.0f));

mvp[0] = 1.0f / px;
mvp[1] = 0;
mvp[2] = 0;
mvp[3] = 0;

mvp[4] = 0;
mvp[5] = -1.0f / py;
mvp[6] = 0;
mvp[7] = 0;

mvp[8] = 0;
mvp[9] =  0;
mvp[10] = m10;
mvp[11] = -1.0f;

mvp[12] = 0;
mvp[13] = 0;
mvp[14] = zNear * m10;
mvp[15] = 0;

I understand that somehow the z axis is inverted, but at this point I don't know how to solve this.

Solution

To flip the matrix in the Y axis:

mvp[5] = -mvp[5];
mvp[7] = -mvp[7];

Then, culling order can be inverted in VkPipelineRasterizationStateCreateInfo struct:

// with frontFace field, replace:
createInfo.frontFace = VK_FRONT_FACE_CLOCKWISE;
// by:
createInfo.frontFace = VK_FRONT_FACE_COUNTER_CLOCKWISE;

// OR with cullMode field, replace:
createInfo.cullMode = VK_CULL_MODE_FRONT_BIT;
// by:
createInfo.cullMode = VK_CULL_MODE_BACK_BIT;