returning PyBytes from async function in pyo3 and pyo3_asyncio

I have the following working implementation for reading bytes from a datachannel and returning the bytes as a list in the python interop:

    fn read<'a>(&self, py: Python<'a>) -> PyResult<&'a PyAny> {
        let mut dc = self.datachannel.clone();
        let mut buf = vec![0u8; 2048 as usize];
        
        pyo3_asyncio::tokio::future_into_py(py, async move {
            let n = dc.read(&mut buf).await.unwrap();
            assert!(n < 2048);
            Ok(buf[..n].to_vec())
        })
    }

However, I would like this function to return the raw bytes as PyBytes instead. My initial thought was to do as the following

    fn read<'a>(&self, py: Python<'a>) -> PyResult<&'a PyAny> {
        let mut dc = self.datachannel.clone();
        let mut buf = vec![0u8; 2048 as usize];
        
        pyo3_asyncio::tokio::future_into_py(py, async move {
            let n = dc.read(&mut buf).await.unwrap();
            assert!(n < 2048);
            let py_bytes = PyBytes::new(py, &buf[..n]);
            Ok(py_bytes.into())
        })
    }

but I am getting the compile error:

37  |           pyo3_asyncio::tokio::future_into_py(py, async move {
    |  _________-----------------------------------_____^
    | |         |
    | |         required by a bound introduced by this call
38  | |             let n = dc.read(&mut buf).await.unwrap();
39  | |             assert!(n < 2048);
40  | |             let py_bytes = PyBytes::new(py, &buf[..n]);
41  | |             Ok(py_bytes.into())
42  | |         })
    | |_________^ `*mut pyo3::Python<'static>` cannot be shared between threads safely

How do I make the read function return PyBytes?

Keeping a handle to py means holding the global interpreter lock (GIL) which you should avoid with asynchronous tasks or else they lose their cooperative nature.

Instead you should use Python::with_gil to reacquire the GIL to construct the result. To return a PyBytes from this, you must un-link it from the new py reference which you can do by turning it into a PyObject.

Give this a try:

use pyo3::ToPyObject;

fn read<'a>(&self, py: Python<'a>) -> PyResult<&'a PyAny> {
    let mut dc = self.datachannel.clone();
    let mut buf = vec![0u8; 2048 as usize];
    
    pyo3_asyncio::tokio::future_into_py(py, async move {
        let n = dc.read(&mut buf).await.unwrap();
        assert!(n < 2048);
        Python::with_gil(|py| Ok(PyBytes::new(py, &buf[..n]).to_object(py)))
    })
}