I have a pipeline in Vulkan with a colorBlendState with blendEnable = false, logicOpEnable = true, and logicOp = XOR so that the output from the fragment shader is XORed with the current contents of the color attachment. I have this working and it behaves as expected... mostly.
The issue is that when overlapping triangles are used in the fragment buffer, the triangles seem to be XORed with each other even though they are all in the same render pass. After researching a bit, it seems like this may be the correct behavior, but it's not what's desired for my case. Is there any way I can revise my pipeline/render pass to only perform the logic op between the fragment shader output and the loaded color attachment contents, and not between individual triangles from the fragment shader?
That is to say, I want all of the triangles defined in the vertex buffer to be drawn without blending or logic op, until the final result is written to the color attachment. Is there any way to do this without adding a second render pass or a subpass?
Logical operations do not apply between triangles. They're relational operations between a fragment (perhaps generated from a triangle) and whatever is in the framebuffer at the time. Which "triangle" generated the framebuffer value is not relevant to the operation.
However, you can employ other operations to prevent overlap. For example, you can use stencil tests. If you clear the stencil bits to a known value, you can have each triangle do a test for that value. The per-fragment operations will only proceed if the stencil bit tests positively against the value in the framebuffer. However, the triangle will also write a different value to the stencil buffer. That way, if two triangles overlap, all fragments from the later one that are on top of the previous one will be culled.
This includes the result of any logical operations.
However, you will have to sort your triangles front-to-back to make this work.