I'm writing shader for unity3d. The shader uses multiple render targets to render post processing effect.
However, I've run into interesting issue.
When Unity3d runs in direct3d mode, by default all standard shaders write data only into first color buffer (i.e. with index 0). I.e. if I attach 3 color buffers to camera, call Camera.Render color buffer with index 0 will contain rendered scene, and all the other buffers will remain untouched unless some shader specifically write in them. My shader utilizes that behavior (I use buffers with indexes 1 and 2 to accumulate data needed for post process effect).
However, in OpenGL mode standard unity3d shaders write in ALL color buffers at once. I.e. if I attach multiple render buffers to a camera, call Camera.Render all 3 buffers will contain copy of rendered scene.
That breaks my shader in OpenGL mode.
How can I fix that? I need to render the whole scene in one go, and only objects that have specific shader should modify additional color buffers.
I need to render scene in one go because using layer masks causes unity to recalculate projector shadows for ALL lights and I need shadows to be correct.
Advice?
Sadly, it turned out that "not writing into one of the render targets" is undocumented behavior in opengl. Standard unity shader when compiled for forward rendering path produces gl_FragData[0] = ...; assignment and writes into only one buffer, which triggers undocumented behavior and causes the mess.
In order to fix that problem, I would need to make unity write data explicitly into additional render targets in standard shaders. Unfortunately, this cannot be done, because there is no "entry point" to "hook" standard shader and write additional data into other color buffers. The closest thing to that is "finalcolor" modifier, but it does not actually allow to write into additional buffers via CG shader (that requires additional data to be from fragment shader, which is inaccessible from surface shader), it is only possible to modify one color.
I decided to rewrite portion of the shader (so it won't trigger undocumented behavior in OpenGL) and gave up on having unity shadowmap support in the effect. As far as I know, there is no other options short of modifying unity engine (requires "special arrangements" and source code access) or replacing entire lighting system with my own.