Why does Azure Self-hosted agent receive Connection refused error when connecting with EventHubs?

When using Azure self-hosted agent deployed on Kubernetes I face the following error:

[Error]2024-06-14T11:28:08.719 [EventHubSendFailed], exception: System.Net.Sockets.SocketException (111): Connection refused
   at Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.CreateConnectionAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.CreateLinkAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
   at Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas, String partitionKey)
   at Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas, String partitionKey)
   at Microsoft.Azure.EventHubs.EventDataSender.SendAsync(IEnumerable`1 eventDatas, String partitionKey)
   at Microsoft.Azure.EventHubs.EventHubClient.SendAsync(IEnumerable`1 eventDatas, String partitionKey)
   at Gateway.EventHub.EventHubSender.ProcessResult(Task task, Object data) in D:\a\1\s\Proxy\Gateway.EventHub\EventHubSender.cs:line 131, source: EventHubSender#xxx

I have the following API policy in place:

<policies>
    <inbound>
        <log-to-eventhub logger-id="xxx">@{
          return new JObject(
              new JProperty("EventTime", DateTime.UtcNow.ToString()),
              new JProperty("ServiceName", context.Deployment.ServiceName),
              new JProperty("RequestId", context.RequestId),
              new JProperty("RequestIp", context.Request.IpAddress),
              new JProperty("OperationName", context.Operation.Name)
          ).ToString();
      }</log-to-eventhub>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

The connection of the api instance with EventHub is configured with API logger + connection string.

The same configuration is working with Azure Managed Gateway.

I tried to deploy the self-hosted gateway on k8s, I expected to see api requests on EventHubs

SocketException (111): Connection refused

The error indicates that a firewall is actively blocking the outbound connection from the pod (or rather, the node) to the Event Hub.

If the k8s cluster is in Azure, it's likely your network infrastructure has Azure Firewall or some other network virtual appliance (NVA). It would not be caused by network security group (NSG), since an NSG silently drops packets if denied, which would precipitate a timeout error instead.

If your cluster is on-premise, there will likely be a firewall which needs a rule to allow outbound connections.

In both cases, the self-hosted gateway will be atrempting an outbound connection to TCP 5671 (and less likely 5672). TCP 5671 connections are wrapped with TLS for security, while 5672 will attempt an upgrade to TLS, but ultimately will accept plain communication.

Azure Event Hubs does support a fallback option of AMQP over WebSockets (TCP 443). This is effectively a TLS connection, just like 5671.

The client (self-hosted gateway) should attempt to fallback to WebSockets. There is no way which I am aware of to explicitly set this as the transport method; It's really down to the client to do it. And, I am simply unsure as to whether the self-hosted gateway AMQP client can.

Assuming your gateway has successfully connected to your APIM instance (which it does on TCP 443), it is likely that it is the AMQP ports 5671 and 5672 which need to be allowed on the firewall.