Python unittest freezes after successful completion of test at sys.exit(0) when using pybind11

I am currently using pybind11 to generate cpp-python interfaces. I have a callback function as follows

pybind_callback.h

#pragma once


#include <stdint.h>

#include <functional>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>

namespace py = pybind11;

typedef std::function<void(uint32_t, uint32_t)> tCbType;

void pybind_add_cb(tCbType func);

pybind_callback.cpp

#include "pybind_callback.h"
#include <vector>

static std::vector<tCbType> cb_func{};

void pybind_add_cb(tCbType func)
{
    cb_func.push_back(func);
}

My pybind11 bindings are implemented as follows

main.cpp

#include <pybind11/pybind11.h>

#include "pybind_callback.h"

namespace py = pybind11;


PYBIND11_MODULE(pybind_interface, m) {
    m.def("pybind_add_cb",            &pybind_add_cb,            "");
}

The generated file is pybind_callback.so which I import for usage in my python unittest which is as follows

test.py

import unittest

from pybind_interface import *

class TestPyBind11(unittest.TestCase):
    def setUp(self):
        self.cb_count = 0
        print("Starting test!")
    
    def cb_func(self, start, end):
        print(f"cb_func(start=0x{start:08x}, end=0x{end:08x})")
        self.cb_count += 1
    
    def test_callback(self):
        pybind_add_cb(lambda start,end: self.cb_func(start, end))
#=================================================================
if __name__ == '__main__':
    unittest.main()

Though the test runs successfully and returns an "OK" after the test has passed, python gets stuck in the sys.exit(not self.result.wasSuccessful()) line in the main.py of the unittest framework. I have verified that the self.result.wasSuccessful() returns 1, and it should get a sys.exit(0) which should then gracefully exit. But this does not happen. I get the results as follows

Starting test!
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
*stuck at this point*

But python never exits in the above case and is stuck at the sys.exit(0) call. If I forcibly change sys.exit(0) to os._exit(0) in the unittest function, then it correctly exits. This is my entire code. This implementation is possible via SWIG but I don't want to use it as the syntax is difficult.

Does anyone know why this happens?

I expect unittest to gracefully exit on encountering sys.exit(0). But this does not happen even when the test passes

Solution

Very related question, maybe a duplicate, however when I run your code on my machine (different optimization levels, with/without valrgind), it looks like double-free issue, and consequently Undefined Behaviour, rather then simply python interpreter freeze.

Apparently on interpreter termination python is trying to "delete" the lambda object previously passed to pybind_add_cb. This happens after it was already deleted on the C++ side during cb_func destruction. To workaround this issue you could provide a cleanup function, like in the quoted answer:

  m.def("pybind_add_cb", &pybind_add_cb, "");
  m.add_object("_cleanup", py::capsule([]{ cleanup(); }));

it can be as simple as

void cleanup() {
    std::cout << "cleanup()\n";
    cb_func.clear();
}