I'm working on a thermal application in Optix, I want to import a GLTF file and then start rays from each primitive.
I don't fully understood the documentation regrading "front" and "back" for faces stored in gltf. Is there a correct way to calculate the outward facing normal using the three vertexes of a triangle and using the cross-product on the sides? If my code works as intended I sometimes get an inward and sometimes an outward normal. Or is the order in which the verteces are stored arbitrary and thus the orientation might inverse? Also for each face there are 3 "normals" stored, if I understand that correctly. If all my faces are flat/not curved, should I be able to use any of these without even having to compute anything?
Thanks for your help in advance!
In glTF, normals are stored per vertex. The normals for faces are typically calculated during rasterization (hardware) in most graphics pipelines, as a linear interpolation between vertex normals for that face.
For what it's worth, glTF does specify a counter-clockwise rotation in primitive.indices. However, materials in glTF are allowed to be double-sided, and the specification for Double Sided indicates that when viewing the back face of a surface, one must flip the normal before evaluating the lighting.
In practice, sometimes authoring tools produce polygon meshes that are inside-out, or even have thin-walled (non-manifold, or non-water-tight) geometry. In such cases it can be difficult to say what is "inside" vs. "outside" as they look the same, and the winding order may be arbitrary. (In Blender Edit Mode, click menu Mesh -> Normals -> Recalculate Outside to fix this).
For single-sided materials, the winding order should not be arbitrary, as the back sides are hidden from view. One would expect counter-clockwise winding order triangles when using those materials. In Blender, this can be done by selecting the "Eevee" engine, bringing up the material properties panel, and putting a checkmark on "Backface culling". Make sure this happens in the material settings, NOT in the viewport settings, otherwise the glTF exporter won't find it. Use "Material Preview" viewing mode to see the result.
If all my faces are flat/not curved, should I be able to use any of these without even having to compute anything?
Yes, if you have completely flat faces, I would expect the vertex normals within each face to be equal, such that linear interpolation would not produce any changes to the normal vector.