python-3.xpython-asynciopikaaio

Random connection errors with aio_pika after 2 days of running


I have asyncio script which connects to rabbitmq with aio_pika library every 40 seconds and checks if there are any messages and prints them out which then repeats forever. However, usually, after 2 or so days of running, I will start receiving endless connection exception errors which will only be solved by restarting the script. Perhaps there are some obvious mistakes in the logic of my asyncio script which I am missing?

#!/usr/bin/python3
import time
import async_timeout
import asyncio
import aio_pika

async def got_message(message: aio_pika.IncomingMessage):
    with message.process():
        print(message.body.decode())

async def main(loop):
    try:
        with async_timeout.timeout(10):
            connection = await aio_pika.connect_robust(
                host='#', 
                virtualhost='#', 
                login='#', 
                password='#',
                port=5671,
                loop=loop, 
                ssl=True
            )

            channel = await connection.channel()

            await channel.set_qos(prefetch_count=100)

            queue_name='mm_message'
            queue = await channel.declare_queue(auto_delete=False, name=queue_name)

            routing_key='mm_msg'
            await queue.bind("amq.topic", routing_key)
            que_len = queue.declaration_result.message_count
            if(que_len > 0):
                await queue.consume(got_message)
    except:
        print("connection problems..")

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    while(True):
        time.sleep(40)
        loop.run_until_complete(main(loop))

This is the error I endlessly receive after some time:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/adapters/base_connection.py", line 364, in _handle_events
    self._handle_read()
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/adapters/base_connection.py", line 415, in _handle_read
    self._on_data_available(data)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1347, in _on_data_available
    self._process_frame(frame_value)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1414, in _process_frame
    if self._process_callbacks(frame_value):
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1384, in _process_callbacks
    frame_value)  # Args
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/callback.py", line 60, in wrapper
    return function(*tuple(args), **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/callback.py", line 92, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/callback.py", line 236, in process
    callback(*args, **keywords)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1332, in _on_connection_tune
    self._send_connection_open()
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1517, in _send_connection_open
    self._on_connection_open, [spec.Connection.OpenOk])
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1501, in _rpc
    self._send_method(channel_number, method_frame)
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1569, in _send_method
    self._send_frame(frame.Method(channel_number, method_frame))
  File "/usr/local/lib/python3.5/dist-packages/aio_pika/pika/connection.py", line 1548, in _send_frame
    raise exceptions.ConnectionClosed
aio_pika.pika.exceptions.ConnectionClosed

Solution

  • except:
        print("connection problems..")
    

    This will catch service Exceptions like KeyboardInterrupt, SystemExit. You should never do such thing if you're not going to reraise exception. At very lest you should write:

    except Exception:
        print("connection problems..")
    

    However in context of asyncio snippet above will break mechanism of cancellation. To avoid it as explained here you should write:

    try:
        await operation
    except asyncio.CancelledError:
        raise
    except Exception:
        log.log('an error has occurred')
    

    Not less important is to understand that connection should not only be opened, but closed also (regardless of what happened between opening and closing). To achieve that people usually use context managers (and in asyncio - asynchronous context managers).

    aio_pika doesn't seem to be exception. As example shows you should use async with when dealing with connection:

    connection = await aio_pika.connect_robust(
        "amqp://guest:guest@127.0.0.1/", loop=loop
    )
    
    async with connection:
        # ...