How to extend SWIG's userdata in Lua code?

I am using SWIG to bind C++ code to Lua. So far it looks good, but now I need to "cheat" and extend a single userdata from within Lua, adding custom fields and methods, etc.

I can't find a way to accomplish it while working within SWIG's directives. I know where in the wrapper code the magic takes place, but I don't fully understand how the __index and __newindex work. Also, SWIG uses __setitem and __getitem which is commented as "/* NEW: looks for the __setitem() fn this is a user provided set fn */" - don't know what that means either. Lastly, my environment automatically calls the script to bind SWIG directives to C++ wrappers before each build, so modifying the sources afterwards is very tedious if I ever choose to recompile or add more Lua bindings.

So far my only conclusion is to go with toLua++, which has this feature documented. Please help me avoid a lot of transition work!

EDIT: I also found that "lua_setuservalue" API call in 5.2 - it seems useful but I don't understand where I would call it among all SWIG binding code.

EDIT: To clear things up:

I create objects by having C function

SoundObj *loadSound(const char *name)
{
    return g_audio->loadSound(name); // Returns SoundObj pointer
}

This function is bound by SWIG by including it's prototype in the *.i swig definition.

In Lua I write code like this:

sound = audio.loadSound("beep.mp3")
sound.scale = 0.2
sound:play()

EDIT: Advancement! I followed Schollii's instructions and wrote the following Lua code. Here, "ground" is the userdata that was acquired earlier using bound C++ code. The code stores the swig's version of __index and __newindex, and then I re-create these functions that query another ("_other") table first.

My current problem here is that the new values stored in "_other" table are shared between all userdata objects of that type. In other words, if ground.nameFoo = "ha!", all other objects have nameFoo field storing "ha!". How do I fix this?

mt = getmetatable(ground)
mt._other = {}
mt.__oldindex = mt.__index
mt.__oldnewindex = mt.__newindex

mt.__index = function(tbl, key)
    if mt.__oldindex(tbl, key) == nil then
        if mt._other[key] ~= nil then
            return mt._other[key]
        end
    else
        return mt.__oldindex(tbl, key)
    end
end

mt.__newindex = function(tbl, key, val)
    if mt.__oldnewindex(tbl, key, val) == nil then
        mt._other[key] = val
    end
end

EDIT: I implemented the solution from this answer: Add members dynamically to a class using Lua + SWIG

The problem is that now my object type is no longer userdata, it is a table. Which means I can no longer organically pass it as an argument to other bound C++ functions that take userdata as argument. Any solution to this? AND I have to do this to every function that returns a userdata object.

Solution

I have crafted a solution, following Schollii's example... but fewer sacrifices. This does not turn your userdata into table, so you still have userdata type (I can pass it to other C++ functions that take userdata). Schollii's solution effectively turned userdata into a table wrapper.

So the first line uses the variable "ground" - this is a userdata instance wrapped by SWIG. Doing this at the beginning of the execution alters the metatable of "ground" userdata type for all instances of that type, but each instance retains a personal table indexed by userdata's memory location. This table lives in _G._udTableReg table.

-- Store the metatable and move the index and newindex elsewhere
mt = getmetatable(ground)
mt.__oldindex = mt.__index
mt.__oldnewindex = mt.__newindex

-- Store the global registry of tables associated with userdata, make a pointer to it in the metatable
_G._udTableReg = {}
mt._udTableReg = _G._udTableReg

-- Rewrite the new index function that looks in the udTableReg using 'self' as index before proceeding to use the old index as backup
mt.__index = function(self, key)
    local ret;
    local privateTable = mt._udTableReg[self]

    -- If the private table exists and has key, return key
    if privateTable ~= nil and privateTable[key] ~= nil then
        ret = privateTable[key]
    -- Use the old index to retrieve the original metatable value
    else ret = mt.__oldindex(self, key) end

    if ret == nil then return 0
    else return ret end
end

-- Try to assign value using the original newindex, and if that fails - store the value in
mt.__newindex = function(self, key, val)
    -- If old newindex assignment didn't work
    if mt.__oldnewindex(self, key, val) == nil then
        -- Check to see if the custom table for this userdata exists, and if not - create it
        if mt._udTableReg[self] == nil then
            mt._udTableReg[self] = {}
        end
        -- Perform the assignment
        mt._udTableReg[self][key] = val
    end
end

I still have not figured out the best place to put this Lua code, or how to get a more elegant way of grabbing the userdata's metatable, without actually using an existing variable.