I am writing a 3D game using OpenGL. I have an sorting problem when rendering translucent triangles. I know that they need to be sorted, but how? I don't want to use BSP trees(if I want to change the position of the object then I need to rebuild the tree.) and OIT algorithms(Performance expensive). I want to sort them by "coverage".

I have two translucent triangles and I use perspective camera to project them onto the screen.

Viewport: 0, 0, 1280, 720

For example

[CORRECT SORTING] a black triangle covers the green triangle.

[INCORRECT SORTING] a green triangle covers the black triangle.

The triangles in the images are opaque in order to clearly see the sorting artifacts.

---

My solution to this problem

Transform the vertices of these triangles to screen space coordinates and compare them by interpolating z coordinate, BUT on my images you can see that one coordinate of black triangle is outside the viewport and therefore this coordinate is weird(something like this: -2896; 1423; 1.169) and I cannot interpolate z coordinate on the black triangle.

How I transform to screen space coordinates

1. Transform triangle vertices by MVP matrix
2. Perform perspective divide
3. Transform to screen space

Transform by MVP matrix

public Vector4f transform(Vector4f vector) {
    float x = m00 * vector.x + m10 * vector.y + m20 * vector.z + m30 * vector.w;
    float y = m01 * vector.x + m11 * vector.y + m21 * vector.z + m31 * vector.w;
    float z = m02 * vector.x + m12 * vector.y + m22 * vector.z + m32 * vector.w;
    float w = m03 * vector.x + m13 * vector.y + m23 * vector.z + m33 * vector.w;

    return new Vector4f(x, y, z, w);
}

Perspective divide

public Vector3f perspectiveDivide() {
    float wInverse = 1.0f / w;
    return new Vector3f(x * wInverse, y * wInverse, z * wInverse);
}

Transform to screen space

public void toScreenCoords(int width, int height) {
    x = (x * 0.5f + 0.5f) * width;
    y = (y * 0.5f + 0.5f) * height;
    z = (z + 1.0f) * 0.5f;
}

How I perform that

Vector4f clipspace = mvp.transform(triangle_vertex);
Vector3f ndc = clipspace.perspectiveDivide();
Vector3f screenspace = ndc.toScreenCoords(1280, 720);

---

What I am doing wrong? Why coordinate that outside the viewport is weird? How to get correct coordinate that outside the viewport?

---

--- EDIT ---

I found out that if one coordinate of the triangle outside the viewport then the triangle must be divided into two.

But how to divide the triangle into two using the clip space coordinates and the clipping planes?

The fast and easy solution to sort (translucent)triangles correctly

"Fake projection sorting" as I call it. If you can assume that the triangles will not intersect, then you can just check if triangle(black or green) intersects a tetrahedron(coordinates from black of green triangle and camera position).

---

An example

First triangle coordinates:

0.0, 0.0, 5.0
0.0, 5.0, 5.0
5.0, 5.0, 5.0

Second triangle coordinates:

0.0, 0.0, 5.0
5.0, 0.0, 5.0
5.0, 0.0, 5.0

Camera position

1.15, 5.8, 5.5

Then, tetrahedron coordinates will be coordinates from a triangle(for example, first) and camera position.

0.0,  0.0, 5.0  // First coordinate from the first triangle
0.0,  5.0, 5.0  // Second coordinate from the first triangle
5.0,  5.0, 5.0  // Third coordinate from the first triangle
1.15, 5.8, 5.5  // Camera position

The last thing you need to do is check if second triangle intersects the tetrahedron. If the second triangle intersects, then it covering first triangle. If you use coordinates from the second triangle to create tetrahedron, then you need to check first triangle for intersection instead of second.

---

Note

Do not use TreeSet or TreeMap from Java to compare triangles using "Fake projection sorting", it can produce incorrect sorting.