My goal

Get getmetatable to return the return value of the function assigned to the __metatable field.

Code:

local x, m = {}, {__metatable = function() return nil end};
setmetatable(x, m);
io.write("Let's get the metatable:", tostring(getmetatable(x)), "\n");

But I'm getting the actual function rather than return.

So how do I get it to be called so I can get nil? So it seems like it has no metatable?

Just rewrite getmetatable to work the way you want it to ;)

do local getmetatable = getmetatable
   function _G.getmetatable(tab)
      local meta = getmetatable(tab)
      if type(meta)=='function' then
         return meta(tab)
      else
         return meta
      end
   end
end

Alternatively, you could just set __metatable to false. This would work for code written like if getmetatable(foo) then, but would break to code like if getmetatable(foo) == false. Arguably, the first is the one you should use, but there's likely someone out there doing the second one.

That'd also hint to the user that there is a metatable, it's just none of their busyness to mess with it.