[SOLVED] linking .s files in windows x86

linking .s files in windows x86_64 C++

I'm able to link my aes-x86_64.s in linux using gcc perfectly fine but not able to in windows using cl, or ld or gcc from TDM.

The assembly file has functions like AES_cbc_encrypt for encryption/decryption from openssl. I'm not going to be using their libraries. I am just interested in linking it. Alternatively, I converted the .s file to masm using their perl script here

https://github.com/openssl/openssl/blob/master/crypto/perlasm/x86_64-xlate.pl

This perl script executes when you build openssl, it basically generates the aes-x86_64.s file which is then built into libcrypto.lib. The issue with generating a masm version is that it adds the .notes.gnu.properties SEGMENT which I end up deleting and compiles but my masm program just doesn't work properly when I call AES_set_encrypt_key which I believe I am calling correctly. Here is the masm program

.data
ks DQ 32 DUP(0)                 ; used for aes_key struct = 60 ints = 240 bytes + int rounds = 244 bytes but rounded off to 256 anyway
OPENSSL_ia32cap_P DQ 2 DUP (0)   ; unsigned int OPENSSL_ia32cap_P[4];

key db 0E0h,0E6h,00Ch,086h,0F1h,010h,0F5h,03Ah,04Eh,065h,088h,079h,074h,0B1h,04Eh,03Dh,006h,022h,09Dh,0B9h,04Eh,026h,025h,0BEh,0D1h,018h,02Bh,056h,073h,0ABh,044h,0F3h,0

.code
include aes-x86_64.asm
externdef ExitProcess:proc


main proc

    sub rsp, 28h                    ;reserve stack space for called functions
    and rsp, 0fffffffffffffff0h     ;make sure stack 16-byte aligned 


    ; AES_set_encrypt_key(key, 256, &ks);
    xor rdx, rdx
    lea r8, ks
    mov rdx, 256
    lea rcx, key
    sub rsp, 20h
    call AES_set_encrypt_key

    exit:
    sub rsp, 20h
    call ExitProcess

main endp

end

the script is too long to paste here but here it is: this is the converted format with the masm syntax. the original is GAS

https://pastebin.pl/view/raw/c8b99bd8

i'm not sure if this MASM syntax is correct but it compiles fine but only if i remove the very last SEGMENT part it creates which is this:

.text$  ENDS
".note.gnu.property"    SEGMENT

    DD  1f - 0f
    DD  4f - 1f
    DD  5
0::


DB  047h
DB  04eh
DB  055h
DB  0
1::

    DD  0c0000002h
    DD  3f - 2f
2::
    DD  3
3::

4::

".note.gnu.property"    ENDS

the function fails returning -1. it should return 0. I debugged with x64dbg.

Solution

Linking has been resolved thanks to the comments posted but the code does not work because the entire GAS file is written for linux. I had assumed it was generic, but in 64-bit that can't be possible because of the calling convention. So I wanted to test if in 32-bit it would work since that code is not subject to any calling convention regardless of system since arguments are pushed on the stack.

I spent the day writing a 32-bit linux GAS program with the 32-bit aesni-x86.s file. When I saw it worked, I migrated that to windows and that worked. I could use a 32-bit program if I want but ultimately I want to code in 64-bit so I'm going to have to convert the code to MASM. I just have to write it accepting arguments in a windows calling convention.