I'm working on some code that has be assembly within my Rust project. A coworker and I are discussing the calling conventions and I have been having trouble finding definitive resources for this.
As I understand it, the Rust calling conventions is undefined. So, when calling into assembly I should use the C calling convention to have something known.
I'm working between three functions:
unsafe extern "C" fn f(a) -> b is a naked function that is called only from Rust. It is entirely assembly and assumes parameters and return values are done as in C (On the stack and put in %eax).
Because this is marked as C and called from Rust, the compiler will ensure that wherever it is called, the C calling convention will be used on it.
unsafe extern "C" fn g() is also a naked function. It is never called from Rust. Only sometimes f returns to this function (that is the ret of f will pop the address of g).
While this is not a true C function (I know it is called from f and immediately use registers than f left values in), it will always call back into a rust function. Essentially, all this does is push parameters onto the stack and then call to h (by use of sym h).
unsafe fn h is then the last function here. It is only ever called from g via that sym in the inline assembly.
So, h should expect to be called with C conventions but can return with Rust conventions (in fact h never returns).
My question is then:
Does using call and sym h from my C function, g, alert the Rust compiler to consistently use C calling conventions in h?
I have previously marked h as extern "C" despite it being pure rust code. This, in my mind, would ensure the compiler respects that h's arguments are C style. But, the code seems to work both ways. I don't want this to change if in the future Rust decides to optimize the call into h differently (and incorrectly).
Using call in inline assembly does not guarantee the function will use the C calling convention (in fact, the compiler has no insight into inline assembly at all). You do need to use extern "C". Using Rust calling convention can appear to work (sometimes the calling conventions are equivalent) but it's not guaranteed.
If you pass a function pointers to h(), they also need to be extern "C" even if they are not called from assembly. As far as I can tell, there is no guarantee about the ABI of Rust-calling-convention functions, so you cannot even pass them, which is why the compiler warns you.
If h() does not return, you can mark it as having the return type !, which will make the compiler verify it indeed does not return (and if called from Rust, will also allow it to exploit this information).