I'm using Microsoft Visual Studio Express 2013. I have a solution with 3 projects. 1 static library and two projects that reference the library. When the optimization for ONLY the library is set to /O2 I get some strange bugs in my game. When I disable the optimization, just for the library, it runs exactly as it does in debug. What would be causing this and how could I fix it?
You could have uninitialized variables which are automatically initialized by the compiler in debug mode, and not in release mode.
Additionally you could be facing alignment issue and minor memory overruns which you are protected from in debug mode due to having no optimizations but when optimizations are enabled and your code is aligned differently this could cause issues and undefined behavior.
Try making sure that all your variables are explicitly initialized, and not assume that:
int i;
is the same as int i = 0;