I'm working on a 3D renderer/game engine in Vulkan. I'm currently considering using GPU tessellation for terrain rendering using heightmaps (as well as being interested in how tessellation can be used in general for various LOD solutions) and I have an issue with how tessellation works, that makes it seem utterly unpredictable and hence almost unusable for me, unless I misunderstand how it works. Note that I'm using Vulkan and my issue is relevant specifically to Vulkan (based on Vulkan's specification), but I imagine this might be equally true for other rendering APIs.
For context - the specific case, which I'm considering right now, is generating a terrain mesh by tessellating a quad patch and then offsetting the resulting vertices using a heightmap (classic height-mapping).
Let's consider a simple case of Quad Tessellation, where we tessellate a Quad into 2 triangles. The tessellation may be performed in 2 different ways:
Depending on the way the quad is split into 2 triangles, the resuling geometry will be completely different (for the same points v0..v3). Let's consider, that points v0 and v3 are higher in the 3D space than points v1 and v2. If we perform the split like on the image on the left, the resulting geometry will end up being a sort of trench caving down:
(https://i.sstatic.net/IG0wE.png)
and if we perform the split like on the right, we'll get more of a tent shape spiking up, like so:
(https://i.sstatic.net/P3FT1.png)
So already for such a simple case, the way the triangles are generated within the tessellated patch has a huge effect on the resulting geometry. The effect is even more dramatic for a more complex mesh if we tessellate the patch into many triangles.
Therefore, I fail to see how we can get any predictable results when using tessellation if we don't know the way the patch is going to be subdivided into triangles. The resulting mesh could take completely different shapes based on how the subdivision is performed.
This wouldn't be a problem, if there was any guarantee as to how the subdivision is performed. But according to Vulkan's specification (https://registry.khronos.org/vulkan/specs/1.3/pdf/vkspec.pdf):
The algorithm used to subdivide the rectangular domain in (u,v) space into individual triangles is
implementation-dependent. However, the set of triangles produced will completely cover the
domain, and no portion of the domain will be covered by multiple triangles.
Based on that paragraph, as well as the entire section of Vulkan's specification dedicated to tessellation, it is my understanding, that there's in fact no guarantee as to how the tessellated patch will be divided and it is in fact implementation-dependent, meaning that the tessellation could possibly be performed differently on different GPUs or even on the same GPU for different Vulkan/driver versions.
In case of terrain mesh generation with heightmapping - I can generate all the vertices using GPU tessellation and sample the correct heights from the heightmap and all the vertices will have the correct coordinates, but the actual shape of the terrain will be different depending on how the tessellation connects the vertices into triangles and that is apparently implementation-dependent...
If my understanding of the above is correct - how can tessellation be used at all, if the resulting geometry is so unpredictable? Is there any way to work around this problem? Or is my understanding of the above incorrect?
So already for such a simple case, the way the triangles are generated within the tessellated patch has a huge effect on the resulting geometry. The effect is even more dramatic for a more complex mesh if we tessellate the patch into many triangles.
Actually, the effect is less dramatic for more complex meshes.
The reason the effect is "dramatic" for a single quad is that... it's just a single quad. It's huge on screen, and where that line goes has a massive effect on the resulting geometry. In particular, what matters most is interpolation of parameters. The values you get as values are interpolated across the faces of those polygons will be substantially different.
If you're tessellating it into, say, 1000 triangles, exactly how those triangles are formed doesn't matter as much, because each triangle contributes far fewer pixels in the resulting image. Any small discrepancies in the tessellation won't matter, because the differences themselves will be small.
Essentially, tessellation is meant to be used to add small details and smoothness to meshes. You should not care about the particulars of exactly how it gets triangulated; any triangulation ought to be "good enough" for your needs.
If the difference in triangulation matters to your use case, then you shouldn't be employing tessellation; you should be using hand-crafted LOD models.
