I'm designing simple sandbox to allow users run small lua scripts and being a bit paranoid I'm not satisfied by just putting it in restricted docker container. I wonder, if I set os and io to nil before running the script (i.e. adding these as two first lines) - will it completely prevent user from gaining access to those standard libraries? Are there any unwanted side-effects? (suppose users just need to solve some basic programming exercise, like finding n-th prime etc).
os = nil
io = nil
will it completely prevent user from gaining access to those standard libraries?
No. In particular, load is still available. This will allow loading malicious bytecode.
Likewise, the debug library is still available as an attack vector too. If the debug library is used, many assumptions about Lua don't hold anymore; it can be abused to gain access to otherwise inaccessible variables or to trigger undefined behavior in the interpreter (which may be or may not be exploitable).
Are there any unwanted side-effects?
With your current io.input, io.read, io.lines and io.write will not be available for reading from stdin or writing to stdout (however print will still be available); how are your programs supposed to communicate with the host? Parameters and return values?
Harmless time & date functions like os.clock, os.date, os.difftime will also be inaccessible.