I am having some odd behavior when using virtualalloc. I'm in c++, Visual Studio 2010.
I have two things I want to allocate, and I'm using VirtualAlloc (I have my reasons, irrelevant to the question)
1 - Space to hold a buffer of x86 assembly code
2 - Space to hold the data structure that the x86 code wants
In my code I am doing:
thread_data_t * p_data = (thread_data_t*)VirtualAlloc(NULL, sizeof(thread_data_t), MEM_COMMIT, PAGE_READWRITE);
//set up all the values in the structure
unsigned char* p_function = (unsigned char*)VirtualAlloc(NULL, sizeof(buffer), MEM_COMMIT, PAGE_EXECUTE_READWRITE);
memcpy(p_function, buffer, sizeof(buffer));
CreateThread( 0, (LPTHREAD_START_ROUTINE)p_function, p_data, 0, NULL);
in DEBUG mode: Works fine
in RELEASE mode: The spun up thread receives a null as its input data. Verified through debugging that when I call createThread the pointer is correct
if I switch the VirtualAlloc's around, so that I allocate the function space before the data space, then both DEBUG and RELEASE mode work fine.
Any ideas why? I've verified all my VS build settings are the same between DEBUG/RELEASE
After copying assembly code into a memory buffer, you can't just jump straight into that buffer. You need to flush CPU caches and the like or it will not work. You can use FlushInstructionCache to do this.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms679350%28v=vs.85%29.aspx
It's hard to say exactly why reordering the allocations would fix the issue, but if you copied the instructions into their buffer and then did a lot of work before jumping into the buffer, that would likely improve the odds of "getting away with it," as the CPU caches would have more of an opportunity to get flushed out by other means.