I've made a pin tool to dump CreatFile win32 calls (in my case CreateFileW) and its return values. It looks like this:
/* ... */
VOID Image(IMG img, VOID *v)
{
RTN cfwRtn = RTN_FindByName(img, "CreateFileW");
if (RTN_Valid(cfwRtn))
{
RTN_Open(cfwRtn);
RTN_InsertCall(cfwRtn, IPOINT_BEFORE, (AFUNPTR)CreateFileWArg,
IARG_ADDRINT, "CreateFileW",
IARG_FUNCARG_ENTRYPOINT_VALUE, 0,
IARG_END);
RTN_InsertCall(cfwRtn, IPOINT_AFTER, (AFUNPTR)CreateFileWafter,
IARG_FUNCRET_EXITPOINT_VALUE, IARG_END);
RTN_Close(cfwRtn);
}
}
/* ... */
VOID CreateFileWArg(CHAR * name, wchar_t * filename)
{
TraceFile << name << "(" << filename << ")" << endl;
}
VOID CreateFileWafter(ADDRINT ret)
{
TraceFile << "\tReturned handle: " << ret << endl;
}
It gives interesting results. For instance, on a small program that just opens an existing file and does nothing else, it gives:
CreateFileW(file.txt)
Returned handle: 0
CreateFileW(file.txt)
Returned handle: 0x74
Returned handle: 0x74
Lots of anomalies.
I also tried to instrument a simple c++ program, that directly calls CreateFileW once, the result:
CreateFileW(file.txt)
Returned handle: 0
CreateFileW(file.txt)
Returned handle: 0xffffffff
Returned handle: 0xffffffff
The file i tried to open did not exist, so the return value (-1 == INVALID_HANDLE_VALUE) is correct at least.
Any ideas? Thanks in advance!
Okay, after some time i finally figured out the causes of these problems.
Well, the PIN documentation says:
NOTE: IPOINT_AFTER is implemented by instrumenting each return instruction in a routine. Pin tries to find all return instructions, but success is not guaranteed
If you dump the function's address at the returns, it turns out that 0 is not returned from CreateFileW. It's returned from another function CreateFileW calls into. This erroneous behaviour of PIN can be fixed with wrapping the CreateFileW method in your own version (dump parameters, call the original function, dump the return value).
Turns out that on my system, CreateFileW calls into Kernelbase.dll's function, that has exactly the same name. Since i instrumented routines by their name, this is correct behaviour. Checking the image name against kernel32.dll solved this.