phprabbitmqheartbeatrabbitmq-management

Why is there no heartbeat on RabbitMQ / PHP connection

I'm trying to connect to RabbitMQ from a PHP consumer using PhpAmqpLib. The connection occasionally silently closes. Mean time to failure is around a couple weeks; I have tried to replicate by blocking ports, shutting down the PHP consumer, and force-closing the connection from the RabbitMQ Management UI - but none of those replicates the error:

The error appears to only happen if there are no messages on the queue for an extended period. I assume the problem is that there's no heartbeat, and this seems to be confirmed by the Management UI: enter image description here

Great, so I just need to find out how to turn on heartbeats. According to the RabbitMQ docs, it is sufficient to set a non-zero heartbeat with either the client or the server.

Both have non-zero heartbeats.

Here's the relevant lines from rabbitmq-diagnostics on the server:

# rabbitmq-diagnostics environment | grep -i heartbeat
      {heartbeat_interval,100},
      {heartbeat,60},

Here's an excerpt from the class where I'm creating the connection, adapted from (undocumented) code I inherited:

use PhpAmqpLib\Connection\AMQPConnectionConfig;
use PhpAmqpLib\Connection\AMQPSSLConnection;
use PhpAmqpLib\Connection\AMQPStreamConnection;
...
    private AMQPStreamConnection $connection;
...
            $heartbeat = 20;
            $conconf = new AMQPConnectionConfig();
            $conconf->setHeartbeat($heartbeat);
            $conconf->setKeepalive(true);

            # See
            # https://php-amqplib.github.io/php-amqplib/classes/PhpAmqpLib-Connection-AMQPStreamConnection.html
            $this->connection = new AMQPStreamConnection(
                $this->host,
                $this->port,
                $this->user,
                $this->password,
                '/', # $vhost
                # $insist - insist on connecting to a specified server, default
                # value:
                false, 
                AMQPConnectionConfig::AUTH_AMQPPLAIN, # $login_method, default value
                null, # $ login_response, default value
                'en_US', # $locale, default value
                3.0, # $connection_timeout, default value
                40.0, # $read_write_timeout, supposed to be at least 2*heartbeat?
                null, # $context, default value
                true, # $ keepalive
                null, # $io, default value
                # $heartbeat, heartbeat interval; default is 0 which RabbitMQ
                # recommends against. See 
                # https://www.rabbitmq.com/docs/heartbeats
                $heartbeat,
                0.0, # $channel_rpc_timeout,
                null, # $ssl_protocol ?
                $conconf # $config
            );

It works to consume messages until the connection drops, so it seems like I'm close...

Solution

  • There were several problems, and I had to fix them all before I got confirmation of a heartbeat in the management UI:

    1. My constructor was wrong; what I used above was adapted from this documentation of the PhpAmqpLib's AMQPStreamConnection constructor. Possibly that was a different version of PhpAmqpLib? But using a long list of positional arguments made it hard to notice that. When I found this constructor source code and compared the call in detail, it gave me a couple more things to try.
    2. From the RabbitMQ docs, "The negotiation process works like this: the server will suggest its configurable value and the client will reconcile it with its configured value, and send the result value back". In particular, this means that if the client (PhpAmqpLib in this case) decides not to follow the AMQP/RabbitMQ protocol on heartbeat negotiation, the server won't force it to do so.
    3. This GitHub discussion thread shows that the PhpAmqpLib community was aware heartbeats weren't fully supported, and they made a change in version 2.12.2 to fix that - but heartbeats were still an optional feature. "PHP can't do heartbeats as configured" is why the client was overriding the server (see #2) in a way counter to the RabbitMQ docs. The source code and constructors from that pull request was useful in helping me figure out how to configure my connection.

    Putting it all together, here's the corresponding excerpt of the code after making the necessary changes:

    use Illuminate\Support\Facades\Log;
use PhpAmqpLib\Connection\AMQPConnectionConfig;
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Connection\Heartbeat\PCNTLHeartbeatSender;
...

    private AMQPStreamConnection $connection;
    # For info on PCNTLHeartbeatSender, see
    # https://github.com/php-amqplib/php-amqplib/pull/815
    private PCNTLHeartbeatSender $sender; # Asynchronous AMQP heartbeat-sender
...
            $heartbeat = 20;
            # See
            # https://php-amqplib.github.io/php-amqplib/classes/PhpAmqpLib-Connection-AMQPStreamConnection.html
            # https://github.com/php-amqplib/php-amqplib/blob/2.12.2/PhpAmqpLib/Connection/AMQPStreamConnection.php
            $this->connection = new AMQPStreamConnection(
                $this->host,
                $this->port,
                $this->user,
                $this->password,
                '/', # $vhost
                # $insist - insist on connecting to a specified server, default
                # value:
                false, 
                AMQPConnectionConfig::AUTH_AMQPPLAIN, # $login_method, default value
                null, # $ login_response, default value
                'en_US', # $locale, default value
                3.0, # $connection_timeout, default value
                40.0, # $read_write_timeout, supposed to be at least 2*heartbeat?
                null, # $context, default value
                true, # $ keepalive
                # $heartbeat, heartbeat interval; default is 0 which RabbitMQ
                # recommends against. See 
                # https://www.rabbitmq.com/docs/heartbeats
                $heartbeat,
                0.0, # $channel_rpc_timeout,
                null, # $ssl_protocol ?
            );
            $this->sender = new PCNTLHeartbeatSender($this->connection);
            $this->sender->register();
            Log::debug('Heartbeat values:', [
                "input"=>$heartbeat,
                "connection"=>$this->connection->getHeartbeat(),
            ]);

    Now both logs and the RabbitMQ Management UI confirm I've got the intended 20s heartbeat in place. Because of the issues with replicating the silently-closed-connection bug described above, it'll be some time before I can confirm this is completely fixed, but I'm hopeful.