I'm curently exploring Lua (5.4.7) vm implementation, it's relatively simple but i can't figure our how return works in case of vararg functions. functions with fixed amount of params are quite simple, just put closure and arguments on stack, call it and thats it
// before call
| caller call frame |
[...][closure][arg1][arg2]
// during call
| callee call frame |
[...][closure][arg1][arg2][...]
// after call
| caller call frame |
[...][return1][return2]
so return instruction just copies return value(s) to call frame base
but var arg call is different it uses special opcode (VARARGPREP) at the begining of instruction list
as far as my understanding go it works like this
// before call
| caller call frame |
[...][closure][arg1][arg2][vararg1][...][varargN]
// during call, before VARARGPREP
| callee call frame |
[...][closure][arg1][arg2][vararg1][...][varargN][...]
// during call, after VARARGPREP
// VARARGPREP copies closure and fixed params to top of the stack,
// and shifts base of call frame
// var args can be accesed via (base - varargs amount + index)
| callee call frame |
[...][closure][arg1][arg2][vararg1][...][varargN][closure][arg1][arg2][...]
// after call
| caller call frame |
[...][closure][arg1][arg2][vararg1][...][varargN][return1][return2]
as we can see, I ended up with stack in invalid state, obviously its not working that way, so my question is what actually happens ?
var arg functions in Lua don't use retrun shorthands (RETURN0 and RETURN1), logic of correcting call frame base is present only in full featured RETURN