pythoncompilationmetaprogramming

python compile with multiple functions


(edit: simpler example with functions) I'm having trouble compiling a python AST with functions that call each other. A minimal example:

def main():
    str = """
def fn_2():
    print("got so far")
        
def fn(input):
    fn_2()

fn("abc")

"""    
    mod = compile(str, "testing", 'exec')
    exec(mod)

if __name__ == "__main__":
    main()

Error:

Traceback (most recent call last):
  File "/Users/wvw/git/n3/fun3/python/test_compile.py", line 26, in <module>
    main()
  File "/Users/wvw/git/n3/fun3/python/test_compile.py", line 23, in main
    exec(mod)
  File "testing", line 8, in <module>
  File "testing", line 6, in fn
NameError: name 'fn_2' is not defined

When I add the function to the original file, it works:

def fn_2():
    print("got here")

def main():
    str = """
def fn_2():
    print("got so far")
        
# def fn(input):
#     fn_2()

fn("abc")

"""    
    mod = compile(str, "testing", 'exec')
    exec(mod)

if __name__ == "__main__":
    main()

But, that is not my goal :-) I'm working on a source-to-source compilation project that spews out a number of functions that call each other.

Clearly, there are some inner workings of compile that I am unaware of. I'm hoping someone with metaprogramming experience in python will be able to shed some light!


Solution

  • I can provide my own solution (mostly thanks to this post!)

    def main():
        str = """
    def fn_2():
        print("got so far")
            
    def fn(input):
        fn_2()
    """    
    
        mod_code = compile(str, "<fun3>", "exec")
    
        new_refs = {}
        exec(mod_code, globals(), new_refs)
        # print(new_refs) # find all compiled codes here
        
        for name, code in new_refs.items():
            # register each compiled code as global
            globals()[name] = code
            
        new_refs['fn']("abc")
                
    
    if __name__ == "__main__":
        main()
    

    It seems that each compiled code needs to be separately registered in the global namespace.