Getting reference to userdata in a table using luaL_ref?

First of all, I appologize for not uploading the full code.

I'm trying to convert userdata to a pointer so it can be passed to other lua chunk using lua_rawgeti().

If you see outletRet() function, it checks the type of a returned value, and if it's a userdata, it passes the pointer to other lua chunk by calling outlet_pointer() and it seems to work fine.

And if userdata is inside a table, outletTable() function is called. And if one of its elements is a userdata it also converts userdata to a pointer and then pass it to other chunk by calling outlet_pointer().

However, when luaL_ref(L, LUA_REGISTRYINDEX) is called, it seems like it gets a reference to the whole table and not just userdata inside it.

How can I get the reference to the userdata and not the whole table?

void ofLua::outletTable() //called from outletRet() below
{
    lua_pushvalue(L, -1);
    lua_pushnil(L);
    int ac = 0;
    t_atom *av = static_cast<t_atom *>(getbytes(sizeof(t_atom) * ac));
    while (lua_next(L, -2))
    {
        av = static_cast<t_atom *>(resizebytes(av, sizeof(t_atom) * ac,
                                               sizeof(t_atom) * (ac + 1)));
        if (lua_isboolean(L, -1))
        {
            av[ac].a_type = A_FLOAT;
            av[ac].a_w.w_float = static_cast<t_float>(lua_toboolean(L, -1));
        }
        else if (lua_isnumber(L, -1))
        {
            av[ac].a_type = A_FLOAT;
            av[ac].a_w.w_float = static_cast<t_float>(lua_tonumber(L, -1));
        }
        else if (lua_isstring(L, -1))
        {
            av[ac].a_type = A_SYMBOL;
            av[ac].a_w.w_symbol = gensym(lua_tostring(L, -1));
        }
        else if (lua_isuserdata(L, -1))
        {
            av[ac].a_type = A_POINTER;
        }
        ac++;
        lua_pop(L, 1);
    }
    lua_pop(L, 1);
    const ofeliaIO &io = dataPtr->io;
    if (io.hasMultiControlOutlets)
    {
        int last = (io.numOutlets >= ac ? ac : io.numOutlets) - 1;
        for (int i = last; i >= 0; --i)
        {
            if (av[i].a_type == A_FLOAT)
                outlet_float(io.outlets[i], av[i].a_w.w_float);
            else if (av[i].a_type == A_SYMBOL)
                outlet_symbol(io.outlets[i], av[i].a_w.w_symbol);
            else if (av[i].a_type == A_POINTER)
            {
                userDataRefVec.push_back(luaL_ref(L, LUA_REGISTRYINDEX));
                outlet_pointer(io.outlets[i], reinterpret_cast<t_gpointer *>(&userDataRefVec.back()));
                luaL_unref(L, LUA_REGISTRYINDEX, userDataRefVec.back());
                userDataRefVec.pop_back();
            }
        }
    }
    else
        outlet_list(dataPtr->ob.ob_outlet, &s_list, ac, av);
    freebytes(av, sizeof(t_atom) * ac);
}

void ofLua::outletRet()
{
    const ofeliaIO &io = dataPtr->io;
    if (!io.hasControlOutlet) return;
    if (lua_isnil(L, -1))
        outlet_bang(io.outlets[0]);
    else if (lua_isboolean(L, -1))
        outlet_float(io.outlets[0], static_cast<t_float>(lua_toboolean(L, -1)));
    else if (lua_isnumber(L, -1))
        outlet_float(io.outlets[0], static_cast<t_float>(lua_tonumber(L, -1)));
    else if (lua_isstring(L, -1))
        outlet_symbol(io.outlets[0], gensym(lua_tostring(L, -1)));
    else if (lua_isuserdata(L, -1))
    {
        userDataRefVec.push_back(luaL_ref(L, LUA_REGISTRYINDEX));
        outlet_pointer(io.outlets[0], reinterpret_cast<t_gpointer *>(&userDataRefVec.back()));
        luaL_unref(L, LUA_REGISTRYINDEX, userDataRefVec.back());
        userDataRefVec.pop_back();
    }
    else if (lua_istable(L, -1))
        outletTable();
}

I'm trying to convert userdata to a pointer

Don't. It's not the same, the pointer does not represent the userdata. You can't retrieve Lua userdata object via pointer to its data area.

If you need to manipulate with userdata on native side, save the userdata in registry with luaL_ref(), and use returned integer as a reference. Later you can retrieve that object from registry and get pointer to its data area whenever you need. But don't store just the pointer.

As for the issue with referencing table, the object to be saved in registry with luaL_ref() must be on Lua stack top. But right now the only thing that stays on Lua stack top is the table. You copy that table's value to iterate with while(lua_next()), and anything that's read from the table within that loop is popped from the stack at the end of the loop.

Probably you shouldn't even build the av structure at all, and process the data immediately as you scan the original table. You don't need to ref/unref userdata then. If you really must have that av, then instead of the pointer to userdata's memory save the integer index returned by luaL_ref(). And you should do it in the first loop, not in the second. The second loop then would do lua_rawgeti() to get userdata object, and after processing you can unref it.