How to set search paths allow lua addins as packages

tl;dr: I want to create lua packages with a custom directory name pattern, having problem with search paths.

The problem

I've got an application that I'm wanted to allow the user to write plugins for, following a similar model to Lightroom:

• A default set of plugins are stored in <app data>/plugins/<name>.myplugin
• <name>.myplugin is a directory bundle which may contain a set of scripts, binaries or other resources
• A plugin may export many different functions to the app, via different scripts
• The names of the exported functions are listed in an info.lua file that is read by the app

The problem I'm grappling with is how best to wrap the plugins as packages (modules + submodules) or regular scripts. I envisage that a plugin might include 3rd party modules:

Foo.myplugin/
    info.lua - returns a table with plugin name, version info, list of exported functions, etc
    Foo.lua - defines the main functions exported by this plugin, which calls other scripts:
    UsefulFunctions.lua - used by Foo.lua
    3rdparty/3rdparty.lua - 3rd party module

If I set the package search path, package.path to include

<appdata>/?.myplugin/?.lua

then I can load the package with Foo=require 'Foo'. However, I can't work out how to get submodules loaded. If Foo.lua calls UsefulFunctions=require 'UsefulFunctions' then this load fails because lua's search path tries to look for UsefulFunctions.myplugin/UsefulFunctions.lua. I can't load it with require 'Foo.UsefulFunctions' either, for similar reasons.

Some options:

• One workaround would be to explicitly add each plugin's path to the package path, but this would cause problems if two plugins each contained a submodule of the same name.
• Another option is to write the plugins to use regular lua scripts rather than providing modules, but this still means that the search paths have to be set up inside each plugin.
• A fallback option could be to lose the .myplugin suffix which would simplify the package search path.
• Patch Lua to explicitly support this type of search path

Is there any way of providing the functionality I need?

I'm currently on Lua 5.1. I know 5.2 has more control over package search paths, but I don't think I have the option of updating to it at the moment. I'm also using luabind, though I don't think it is relevant to this.

You could customize the way Lua searches for modules using a custom searcher function, using the mechanisms outlined in the documentation of require and package.loaders.

The trick is to detect that the module can be found in a directory with the .myplugins suffix and to keep track of the path of the bundles. Consider the following scripts.

-- <appdata>/plugins/foo.myplugin/foo.lua

local auxlib = require 'foo.auxlib'
local M = {}
function M.Foobnicator()
    print "Called: Foobnicator!!"
    auxlib.AuxFunction()
end
return M

 

-- <appdata>/plugins/foo.myplugin/auxlib.lua

local M = {}
function M.AuxFunction()
    print "Called: AuxFunction!!"
end
return M

 

-- main.lua

package.path = package.path .. ";" 
    .. [[<appdata>/plugins/?.myplugin/?.lua]]
local bundles = {}  -- holds bundle names and pathnames

local function custom_searcher( module_name )
    if string.match( module_name, '%.' ) then
        -- module name has a dot in it - it is a submodule, 
        -- let's check if it is inside a bundle
        local main_module_name, subname = 
            string.match( module_name, '^([^.]-)%.(.+)' )
        local main_path = bundles[ main_module_name ]
        if main_path then  -- OK, it's a submodule of a known bundle
            local sub_fname = string.gsub( subname, '%.', '/' )
            -- replace main module filename with that of submodule
            local path = string.match( main_path, '^.*[/\\]' ) 
                .. sub_fname .. '.lua'
            return loadfile( path )
        else    -- not a bundle - give up the search
            return
        end
    end

    -- search for the module scanning package.path
    for template in string.gmatch( package.path, '[^;]+' ) do
        if string.match( template, '%.myplugin' ) then -- bundle?                
            local module_path = 
                string.gsub( template, '%?', module_name )
            local fh = io.open( module_path )     -- file exists?
            if fh then  -- module found
                fh:close()
                bundles[ module_name ] = module_path
                return loadfile( module_path )
            end
        end
    end
end

-- sets the custom searcher as the first one so to take
-- precedence over default ones
table.insert( package.loaders, 1, custom_searcher )

local foo = require 'foo'
foo.Foobnicator()

Running main.lua will produce the following output:

Called: Foobnicator!!
Called: AuxFunction!!

I hope this will put you on the right track. Probably it doesn't cover every possibility and the error handling is not at all complete, but it should give you a good base to work on.