[SOLVED] How to work around missing PyModule_Create2 in AMD64 Win Python35

How to work around missing PyModule_Create2 in AMD64 Win Python35_d.lib?

I'm trying to debug an extension module that worked fine in 32 bit Python 2.7, but not so fine in 64 bit Python 3.5.

I've used the AMD64 Web installer from Python.org, and yet in the link I get

__imp_PyModule_Create2 (referenced in libboost_python-vc120-mt-gd-1_57.lib(module.obj))

is unresolved. It's the only symbol that is unresolved.

Is this intentional? I saw an old bug report that seemed to indicate that the Stable ABI was exempt from debug builds. (Which is why I'm posting on SO instead of filing a bug report)

If it is intentional, is it expected that I would link with python35_d.lib and then python35.lib, or is there another way to resolve this?

Solution

I've discovered the issue here; the AMD64 factor is irrelevant.

The key is that the Boost Python library expects to link against the Release version of Python, even if it is the debug version of Boost Python. By trying to link an extension module that depends on Boost Python, Boost's

config/auto_link.hpp

will (by default) create a link dependency -- using

#pragma comment(lib, string_of_library_name)

-- on python35.lib. This is bad news if in the makefile for your extension module you have specified python35_d.lib, with the expectation that your python will be invoked as python35_d.exe.

I found this by running

dumpbin /EXPORTS python35.lib > python35_exp.txt
dumpbin /EXPORTS python35_d.lib > python35_d_exp.txt

and comparing the two. The key difference is that the release version exports the symbols PyModule_Create2 and PyModule_FromDefAndSpec2, whereas the debug version exports PyModule_Create2TraceRefs and PyModule_FromDefAndSpec2TraceRefs. This is an apparent deliberate choice by the Python developers to make sure that debug extension modules only work with debug Python. In the Python source code, one of the first lines in include/object.h is

/* Py_DEBUG implies Py_TRACE_REFS. */
#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS)
#define Py_TRACE_REFS
#endif

The giveaway is in include/modsupport.h

#ifdef Py_TRACE_REFS
 /* When we are tracing reference counts, rename module creation functions so
    modules compiled with incompatible settings will generate a
    link-time error. */
 #define PyModule_Create2 PyModule_Create2TraceRefs
 #define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
#endif

The solution is to build special versions of the Boost libraries that specifically link against python35_d.lib. There are a few steps involved:

Run 'bootstrap.bat --with-python="C:\Program Files\Python35"'
Edit user-config.jam in your home directory so that it looks like (spaces are important)

    using python : 3.5 : C:\\PROGRA~1\\Python35 ;
    using python : 2.7 : C:\\Python27 ;
    using python : 3.5 : C:\\PROGRA~1\\Python35\\python_d
      : # includes
      : # libs
      : <python-debugging>on ;

Invoke b2.exe with the extra option "python-debugging=on", like so

.\b2.exe toolset=msvc-12.0 threading=multi variant=debug address-model=64 --with-python --debug-configuration python-debugging=on stage

Note that to resolve dependencies you are going to have to also compile date_time, thread, chrono, filesystem and system (just replace "--with-python" with "--with-otherlibname".)

The final step is to make sure that the extension module that you are building links against the "right" library. I found that it was sufficient to define the preprocessor symbols BOOST_DEBUG_PYTHON and BOOST_LINKING_PYTHON, as well as the expected _DEBUG because in config/auto_link.hpp there are the lines

    #        if defined(_DEBUG) && defined(BOOST_DEBUG_PYTHON) && defined(BOOST_LINKING_PYTHON)
    #            define BOOST_LIB_RT_OPT "-gyd"
    #        elif defined(_DEBUG)
    #            define BOOST_LIB_RT_OPT "-gd"
    #        else
    #            define BOOST_LIB_RT_OPT
    #        endif

That ought to do it - a debug version of a Python extension module that should compile and link against a Boost Python library that expects to link against python35_d.lib, and which will not crash when loaded by a script invoked with python_d.exe.