Difference behavior of C program compiled with GNU -O2 and -O3

I am working on an embedded Linux system (5.10.24), and there are several applications running in it.

Recently, I found something strange with one C program as follows.
The C program uses APIs provided by library from a SDK, which is built with GNU -O2, but the C program is built with GNU -O3 by default.

Then it is found sometimes (1 in hundreds of running) the C program would trigger Segmentation Fault in a function in the SDK.

If the C program is built with the same GNU -O2 as the SDK, there is NO such Segmentation Fault triggered in the thousands of its running.

I googled the difference of -O2 and -O3, and found someone mentioned that -O3 may trigger weired and funky bug, is it true?

Solution

As a general rule, assume that major software used by millions is functioning correctly and your own software, developed just now by you and used only by you is the cause of the problem. Finding genuine bugs in GCC like this is extremely rare in a developer's carrear, and a bit like winning the lottery. Finding bugs in closed source proprietary code is a bit more common depending on the size of organisation and user base and age of product.

Optimisations do not affect pre-compiled code, only code that you compiled with GCC.

Optimisation levels are a shorthand for adding a whole bunch of optimisation flags. You can experiment with them individually to see which one causes problems. For -O3 that's:

-fgcse-after-reload
-fipa-cp-clone
-floop-interchange
-floop-unroll-and-jam
-fpeel-loops
-fpredictive-commoning
-fsplit-loops
-fsplit-paths
-ftree-loop-distribution
-ftree-partial-pre
-funswitch-loops
-fvect-cost-model=dynamic
-fversion-loops-for-strides

Ususally, mismatching optimisation levels between your code and APIs shouldn't cause a problem. The calling convention is unaffected. In rare cases struct padding can get misaligned but that is rare.

More commonly optimisations do one of two things:

In threaded code, aggressive optimisations can re-order your code in such a way as to no-longer be thread safe. This is usually the result of you, the developer, failing to use keywords such as C's volatile correctly.
In all code, lower levels of optimisation have some addational protection which is helpful for debugging code. These include things like adding padding with pre-defined values that allow a debugger to spot a buffer overrun. Such padding can prevent your program crashing altogether when it should have. Note I don't believe padding is removed by the above optimisations for -O3 over -O2. It may still re-order things such that an existing mistake of buffer overrun or array indexing error now causes a fatal error where it did not origionally.

I've not seen the context of what you've read, but on balance, assume that optimising your code will trigger a bug in your code, not the compiler.