I have been doing a lot of low level coding recently and wanted to try writing a program fully in machine code to further my knowledge (translating down from x64 assembly, and adding it into the correct sections of a PE file)
The only problem I am having is that I am on Windows and I have to use the win API instead of syscalls to have anything I can interact with like a window or even just stdout, and obviously there aren't a lot of guides on how to call a .DLL function from machine code. I'm assuming it's probably done by the OS using some field in the PE file and not actually the compiler. Does anyone have any information on this?
My actual example here: https://github.com/joshudson/dossuperfloppy/blob/660586f2bb3d660d8154b0cb6857562dc4892bcf/postntfs.asm#L374 The differences between 32 bit and 64 bit are small enough that this is still a good example of what the .idata section should look like.
You need to emit the .idata section (yes it can be combined with another section; but the contents need to be there...)
In the PE header, the second RVA pointer points to the import directory, while the 13th points to the import address table (more on that later).
Each entry in the import table consists of five dwords. The first dword is an RVA to the position in the IA table this dll starts. The second and third will be zero in the simplest case. The fourth is an RVA to the ASCIIZ string of the DLL name (no you can't import dlls in unicode names), and the fifth is an RVA to the import addresses translated table. The table is terminated with an entry of five zero dwords. (Why it's ended this way rather than using the size parameter in the header I have no idea; note that the size in the RVA table includes the terminating entry.) The import table must be dword aligned.
The import address table consists of machine-word (dword for 32 bit, qword for 64 bit) RVA pointers into the import names table. A zero stops processing for this particular imported DLL. This table must be machine-word aligned.
The import names table consists of entries that contain a word hint (if the image has not yet been bound, 0 is the best choice for the hint value) followed by an ASCIIZ string that gives the name of the symbol being imported. Each entry in the name table must be word aligned. Failing to word-align entries in the name table results in a particularly confusing error.
In the disk image, the import addresses translated (IAT) table is a copy of the import addresses table. During load, it is overwritten with pointers to the actual imported symbols. Thus, the .idata section is declared writable.
The easiest way to use an imported symbol is to emit a call [rel iat-offset] instruction. Since you are intent on the pain of doing this in machine code, the instruction to emit is encoded as FF 15 followed by a little-endiand dword that is the number of bytes between the end of the (six byte) instruction and the entry in the import address translation table (as loaded in RAM; note that this most likely won't be the distance in the image).
RVA: relative virtual address. A pointer whose base address is the load address of the image in memory.