I am rendering models based on milions (up to ten) of triangles using VBOs and I need to detect which of these triangles the user may click on.
I try to read and understand how the "name stack" and the "unique-color" work. I found that the name stack can contain at max only 128 names, while the unique-color can have up to 2^(8+8+8) = 16777216 possible different colors, but sometimes there could be some approximations, so it can get modified..
Which is the best strategy for my case?
Basically, you have 2 classes of options:
The "unique color way per triangle", which means you attach an id to every triangle, and render out the id's to a seperate render target. It can be 32 bit's (8 for rgb, 8 for a), but you could add a second one for even more ideas. It'll be fiddly to get the id's per triangle, but it's relatively easy to implement. Can be quite detrimental to performance though (fillrate).
Proper ray tracing. You almost certainly want to have an acceleration structure (octree, kd,...), but you probably already have one for frustum culling. One ray really isn't a lot, this method should be very fast.
Hybrid. probably the easiest to implement. Render out the vertex buffer id ("unique color per buffer:), and when you know which vertex buffer was selected", just trace a ray against all the triangles.
In the general case, I would say 2) is the best option. If you want to have something work quickly, go for 3). 1) is probably pretty useless.