pytestpython-asynciopytest-asyncio

speed up test with asyncio


Here is my code

@pytest.mark.asyncio
async def test_async():
    res = []
    for file_id in range(100):
        result = await get_files_async(file_id)
        res.append(result)
    assert res == [200 for _ in range(100)]


async def get_files_async(file_id):
    kwargs = {"verify": False, "cert": (cert, key)}
    resp = requests.request('GET', url, **kwargs)
    return resp.status_code

The timing from pytest shows it takes 118 sec to finish, which is very close to the test that sends request to url in sequence. Is there any improvement that can speed up this test? Thanks.


Solution

  • You cannot speed this up using async, since you are using requests, which is a sync pkg, so each call stops the even loop.

    You can either switch to running the requests in threads or switch to an async pkg like httpx or aiohttp

    If you do switch to a different pkg, change the test_async to this to run the requests in parallel

    @pytest.mark.asyncio
    async def test_async():
        tasks = []
        for file_id in range(100):
            tasks.append(asyncio.create_task(get_files_async(file_id)))
        res = await asyncio.gather(*tasks)
        assert res == [200 for _ in range(100)]