I used #pragma pack(push, 2) at the beginning of a struct in a header file but forgot the corresponding #pragma pack(pop). After including this header file, I included fstream. On creating an ofstream object, I am seeing stack smashing. Details of the exact scenario and code are as follows.
I was following a C++ course and had written a code for the project. My program was crashing due to stack smashing. I tried to look for any obvious overflow errors but couldn't find any. I changed almost all of my code to resemble the code provided by the instructor. The only difference being the order of included header files. I had included my header files followed by fstream while the instructor had included the fstream header file at the top. Still I was getting the same issue. So I changed even the order of header files and voilà, problem disappeared.
As this was strange for me, I tried to localize the issue in my code.
I inserted print statements throughout the code to find the function where stack smashing was happening.
On finding the function I used gdb to set watch for the canary value used by my program to check for stack smashing. I detected stack smashing in the constructor of ofstream object.
At this point I knew that some header file included before fstream was interfering with it. So now I checked through all of my header files for any silly mistakes and found a struct preceded by #pragma pack(push, 2) but not followed by corresponding #pragma pack(pop). This struct was to be written as a binary file. Correcting this resolved the issue.
Since the entire project is irrelevant, I have reproduced the issue with a simple code snippet. Although the problem is resolved, I would like to know why it happened. I understand that pragma pack directive was used to prevent the compiler from inserting paddings inside the struct as the struct had to be written to a binary file. Omitting the pack(pop) at the end of the struct, uses the same for all the subsequent header files. But can this lead to the ofstream constructor writing past the stack frame?
I am using gcc v7.4.0 on Ubuntu 18.04.
/* code.cpp */
#include "header.h"
#include <fstream>
using namespace std;
int main(){
ofstream fout;
fout.open("file.ext", ios::out|ios::binary);
fout.close();
return 0;
}
The header file
#ifndef HEADER_H_
#define HEADER_H_
#pragma pack(push, 2)
struct something{
int a;
};
//#pragma pack(pop)
//Uncommenting the above line solves the problem
#endif
Since pragma pack affects the layout of class instances, your version of ofstream doesn't look the same as the one that was used to compile your standard library. Formally, you have an ODR violation, and that results in undefined behaviour.
In practice, the functions from your C++ runtime are operating blindly on data with the wrong layout, so it only makes sense that fireworks ensue. The stack smashing in particular is expected since the packed class is shorter that the unpacked one, so writing to the end of the instance overflows the space main's stackframe reserved for it.