I am trying to analyze the Excel 4.0 macro sample, introduced at https://outflank.nl/blog/2018/10/06/old-school-evil-excel-4-0-macros-xlm.
While testing, I noticed it does work well on 32-bit Excel, but doesn't work properly on 64-bit.
Eventually, I found the reason and that is: VirtualAlloc function always returns 4-byte out of 8-byte.
Implementation here:
= REGISTER("Kernel32", "VirtualAlloc", "JJJJJ", "valloc", , 1, 9)
= valloc(0, 64 * 1024, 4096, 64)
How to solve it?
You have declared the return type as "J", which is only 32 bits / 4 bytes long.
You may try "D", which is 8 bytes long, and then use a mem copy function to copy it to an 8 byte variable, or you can try putting in a dummy parameter, and try to recover the next 4 bytes from the dummy parameter.
Excel doesn't crash when you declare external calls incorrectly, because Excel does a stack check after each call, and makes sure that the stack is restored correctly.
Or it may be that 64 bit Excel has support for 64 bit longs in XLM macros, using some new type declaration-- I haven't seen that, but I guess it would be worth looking.