Overriding C++ class virtual methods in Pybind11 with trampoline class - but still calling base method from C++

I'm implementing a Pybind11 module, and I'm exposing a C++ class to Python that it's meant to be subclassed. I'm following what's described in here (https://pybind11.readthedocs.io/en/stable/advanced/classes.html) and below is the code for the C++ base and trampoline class.

class CollisionResponse {
public:
    CollisionResponse() : _tag1(0), _tag2(1) {}
    virtual ~CollisionResponse() = default;
    virtual void onStart() const {
        std::cout << "base called\n";
    }

    int getTag1() const;
    int getTag2() const;
protected:
    int _tag1;
    int _tag2;
};


class PyCollisionResponse : public CollisionResponse {
public:
    using CollisionResponse::CollisionResponse;  

    void onStart() const override {

        PYBIND11_OVERRIDE(
                void,                               // Return type
                CollisionResponse,                  // Parent class
                onStart                     // Name of function in C++
        );

    }
};

Below is how I expose the class to Python:

py::class_<CollisionResponse, PyCollisionResponse, 
    std::shared_ptr<CollisionResponse>>(m, "CollisionResponse")
    .def(py::init<>())
    .def("onStart", &CollisionResponse::onStart);

Now I have a C++ object which holds a map of CollisionResponse objects, that are passed from Python via the addResponse method:

class CollisionEngine:
...
    void addResponse(std::shared_ptr<CollisionResponse> response) {
        _response.insert({std::make_pair(response->getTag1(), response->getTag2()), response});

    }

    std::unordered_map<std::pair<int, int>, std::shared_ptr<CollisionResponse>> _response;

Last but not least is the Python code:

class TestCollision(example.CollisionResponse):

    def onStart(self):
        print('derived class')
...

cr = CollisionResponse()
cr.addResponse(TestCollision())

Now, in the CollisionEngine class, I have a method that calls the onStart method from the CollisionResponse objects stored in _response. However, no matter what happens, the base implementation always get called. In debug, I can see that the PyCollisionResponse::onStart is called, but then execution goes inside the CollisionResponse::onStart, and not in the Python TestCollision.onStart.

I'm really struggling figuring out what's wrong!

Solution

The C++ object and the python object are not one object, they are two objects, if you don't declare a holder type then the C++ object will be stored inside the python object, so they should have the same lifetime.

If you do declare a holder (std::shared_ptr) then only the holder will be stored inside the python object, the C++ object will stay alive as long as the python object is alive, but the opposite is not true, if the python object got destroyed the C++ object can stay alive if there is still another shared_ptr to it alive, and when you call a method on it, it will act as if the python override doesn't exist.

what you want is to store the python object too to keep it alive, by storing this py::object somewhere. since you are exposing CollisionEngine to python then maybe you can intercept the call to addResponse (by making it virtual) and store this TestCollision in a list before passing it to C++, you will need to do a similar trick in the unregistration too, (call a python unregistration function with the C++ object before you unregister it from the C++ side, pybind11 should translate it back to the python object in both cases).