How can I let a shared library, called by Python, access the same Python instance's globals?

So I've created this main.py:

#!/usr/bin/python3

import ctypes


foo = 1


# Should print 2, but prints 1
def print_foo():
    global foo
    print(foo)


def main():
    global foo
    foo = 2

    dll = ctypes.PyDLL("./foo.so")

    foo = dll.foo
    foo.restype = None
    foo()


if __name__ == "__main__":
    main()

and this foo.c:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

#include <assert.h>

#define CHECK_PYTHON_ERROR() {\
    if (PyErr_Occurred()) {\
        PyErr_Print();\
        fprintf(stderr, "Error detected in foo.c:%d\n", __LINE__);\
        exit(EXIT_FAILURE);\
    }\
}

void foo(void) {
    PyObject *module = PyImport_ImportModule("main");
    CHECK_PYTHON_ERROR();
    assert(module);

    PyObject *print_foo = PyObject_GetAttrString(module, "print_foo");
    CHECK_PYTHON_ERROR();
    assert(print_foo);

    PyObject *result = PyObject_CallObject(print_foo, NULL);
    CHECK_PYTHON_ERROR();
    assert(result);
}

I've compiled foo.c to foo.so with gcc foo.c -o foo.so -shared -fPIC -Wall -Wextra -Wpedantic -Wfatal-errors -g -lpython3.13 -I/home/trez/.pyenv/versions/3.13.0/include/python3.13 -L/home/trez/.pyenv/versions/3.13.0/lib -ldl -Wl,-rpath,/home/trez/.pyenv/versions/3.13.0/lib -lm, where those last flags came from running python3.13-config --includes --ldflags.

Running python3.13 main.py prints 1. It implies that the foo = 2 statement did not happen in the instance which printed foo, which is bad.

The reason this program currently prints 1, is because the PyImport_ImportModule() (docs) call essentially creates a new environment, rather than reusing the original one.

Changing it to PyImport_AddModule() (docs) or PyImport_AddModuleRef() (docs) prints this:

AttributeError: module 'main' has no attribute 'print_foo'
Error detected in foo.c:20

If we look at PyImport_AddModule its docs, we just see:

Similar to PyImport_AddModuleRef(), but return a borrowed reference.

And if we then look at PyImport_AddModuleRef its docs, there's this snippet:

This function does not load or import the module; if the module wasn’t already loaded, you will get an empty module object. Use PyImport_ImportModule() or one of its variants to import a module.

So this seems to imply that the AttributeError: module 'main' has no attribute 'print_foo' error was caused by the module not being loaded. This doesn't make sense to me though, since using PyImport_AddModule/PyImport_AddModuleRef and adding this after the assert(module); line:

PyObject *g = PyEval_GetGlobals();
CHECK_PYTHON_ERROR();
PyObject *g_repr = PyObject_Repr(g);
CHECK_PYTHON_ERROR();
const char *s = PyUnicode_AsUTF8(g_repr);
CHECK_PYTHON_ERROR();
printf("s: '%s'\n", s);

prints this:

s: '{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x6ffc309a5e00>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': '/home/trez/Programming/ctypes-run-bug/main.py', '__cached__': None, 'ctypes': <module 'ctypes' from '/home/trez/.pyenv/versions/3.13.0/lib/python3.13/ctypes/__init__.py'>, 'foo': <_FuncPtr object at 0x6ffc30946450>, 'print_foo': <function print_foo at 0x6ffc308231a0>, 'main': <function main at 0x6ffc3088f4c0>}'
AttributeError: module 'main' has no attribute 'print_foo'
Error detected in foo.c:28

Note the 'print_foo': <function print_foo at 0x6ffc308231a0> near the end of the first line, implying that the function is found.

My guess as to what's happening here is that these globals are from the original Python instance, but that the PyImport_AddModule("main") call somehow fails to find that instance.

How do I keep main.py untouched, while modifying foo.c in a way that 2 gets printed, instead of 1? Cheers.

Solution

juanpa.arrivillaga gave the great advice to have the C code get the main module by inspecting the C equivalent of sys.modules, and this solution does successfully print 2:

#!/usr/bin/python3

import ctypes


foo = 1


def print_foo():
    global foo
    print(foo)


def main():
    global foo
    foo = 2

    dll = ctypes.PyDLL("./foo.so")

    bar = dll.bar
    bar.restype = None
    bar()


if __name__ == "__main__":
    main()

#define PY_SSIZE_T_CLEAN
#include <Python.h>

#include <assert.h>

#define CHECK_PYTHON_ERROR() {\
    if (PyErr_Occurred()) {\
        PyErr_Print();\
        fprintf(stderr, "Error detected in foo.c:%d\n", __LINE__);\
        exit(EXIT_FAILURE);\
    }\
}

void bar(void) {
    PyObject *modules = PySys_GetObject("modules");
    CHECK_PYTHON_ERROR();
    assert(modules);

    PyObject *main_object = PyDict_GetItemString(modules, "__main__");
    CHECK_PYTHON_ERROR();
    assert(main_object);

    PyObject *print_foo = PyObject_GetAttrString(main_object, "print_foo");
    CHECK_PYTHON_ERROR();
    assert(print_foo);

    PyObject *result = PyObject_CallObject(print_foo, NULL);
    CHECK_PYTHON_ERROR();
    assert(result);
}

MarkTolonen made me aware that I was accidentally overwriting my foo value of 2 with my foo = dll.foo line. I've changed that to bar = dll.bar; bar.restype = None; bar(), and renamed void foo(void) { ... } to void bar(void) { ... }.

As juanpa.arrivillaga mentioned though, if print_foo() and foo = 1 are moved to say helper.py, and I change main() to look like def main(): helper.baz() (where baz() looks like the old main()), then I get AttributeError: module '__main__' has no attribute 'print_foo'. This is fine in my case, since I don't mind hardcoding the module name where print_foo() resides in foo.c.