Is there a way to call require in a Lua file, and have the module set the environment of the file that calls it? For example, if I have a DSL (domain specific language) that defines the functions Root and Sequence defined in a table, can I have something like setfenv(1, dslEnv) in the module that allows me to access those functions like global variables?

The goal I in mind is using this is a behavior tree DSL in a way that makes my definition file look like this (or as close it as possible):

require "behaviortrees"

return Root {
    Sequence {
        Leaf "leafname",
        Leaf "leafname"
    }
}

without having to specifically bring Root, Sequence, and Leaf into scope explicitly or having to qualify names like behaviortrees.Sequence.

In short, I'm trying to make the definition file as clean as possible, without any extraneous lines cluttering the tree definition.

Can I have something like setfenv(1, dslEnv) in the module that allows me to access those functions like global variables?

Sure you can. You just have to figure out the correct stack level to use instead of the 1 in your setfenv call. Usually you'd walk up the stack using a loop with debug.getinfo calls until you find the require function on the stack, and then you move some more until you find the next main chunk (just in case someone calls require in a function). This is the stack level you'd have to use with setfenv. But may I suggest a ...

Different Approach

require in Lua is pluggable. You can add a function (called a searcher) to the package.loaders array, and require will call it when it tries to load a module. Let's suppose all your DSL files have a .bt suffix instead of the usual .lua. You'd then use a reimplementation of the normal Lua searcher with the differences that you'd look for .bt files instead of .lua files, and that you'd call setfenv on the function returned by loadfile. Something like this:

local function Root( x ) return x end
local function Sequence( x ) return x end
local function Leaf( x ) return x end


local delim = package.config:match( "^(.-)\n" ):gsub( "%%", "%%%%" )

local function searchpath( name, path )
  local pname = name:gsub( "%.", delim ):gsub( "%%", "%%%%" )
  local msg = {}
  for subpath in path:gmatch( "[^;]+" ) do
    local fpath = subpath:gsub( "%?", pname ):gsub("%.lua$", ".bt") -- replace suffix
    local f = io.open( fpath, "r" )
    if f then
      f:close()
      return fpath
    end
    msg[ #msg+1 ] = "\n\tno file '"..fpath.."'"
  end
  return nil, table.concat( msg )
end


local function bt_searcher( modname )
  assert( type( modname ) == "string" )
  local filename, msg = searchpath( modname, package.path )
  if not filename then
    return msg
  end
  local env = { -- create custom environment
    Root = Root,
    Sequence = Sequence,
    Leaf = Leaf,
  }
  local mod, msg = loadfile( filename )
  if not mod then
    error( "error loading module '"..modname.."' from file '"..filename..
           "':\n\t"..msg, 0 )
  end
  setfenv( mod, env ) -- set custom environment
  return mod, filename
end


table.insert( package.loaders, bt_searcher )

If you put this in a module and require it once from your main program, you can then require your DSL files with the custom environment from .bt files somewhere where you would put your .lua files as well. And you don't even need the require("behaviortrees") in your DSL files. E.g.:

File xxx.bt:

return Root {
  Sequence {
    Leaf "leafname",
    Leaf "leafname"
  }
}

File main.lua:

#!/usr/bin/lua5.1
require( "behaviortrees" ) -- loads the Lua module above and adds to package.loaders
print( require( "xxx" ) ) -- loads xxx.bt (but an xxx Lua module would still take precedence)