Configure Max HTTP Connections on Spring Cloud Gateway MVC Route

I have an API Gateway server written with Sprint Boot 3 and Spring Cloud Gateway MVC (Spring Cloud 2024.0.1). I am using Eureka for API Discovery (spring-cloud-starter-netflix-eureka-client), which brings in Apache HTTP Client 4 & 5.

The routes are configured:

@Bean
public RouterFunction<ServerResponse> sampleRoute() {
    return RouterFunctions
        .route()
        .route(GatewayRequestPredicates.path("/sample-api/a-service/**"), HandlerFunctions.http())
        .before(BeforeFilterFunctions.routeId("sample-route"))
        .before(BeforeFilterFunctions.stripPrefix(2)) // remove "/sample-api/a-service/"
        .filter(RetryFilterFunctions.retry((r) -> r.setRetries(3).setMethods(Set.of(HttpMethod.GET, HttpMethod.OPTIONS, HttpMethod.HEAD))))
        .filter(LoadBalancerFilterFunctions.lb("A-SERVICE"))
        .filter(CircuitBreakerFilterFunctions.circuitBreaker("aServiceCircuitBreaker"))
        .build();
}

This has been working fine until we began testing the gateway under load. Unfortunately under load I am seeing the gateway hang as it exhausts all available HTTP connections for multiple routes. We see the following in the logs when the gateway becomes overwhelmed:

DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.get-ex-0000376182 endpoint leased [route: {}->[http://example.org:9080]][total available: 9; route allocated: 5 of 5; total allocated: 20 of 25]

The values of 5 max connections per route and 25 total max connections match the defaults on org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.

How do I override these default max connection settings so that I can increase the max connections overall and the max connections per route?

The following code customizes the number of HTTP connections when using Apache HTTP client 5 with Spring Cloud Gateway:

@Bean
public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() {
    // increase the size of the http connection pool for the API gateway routes
    return PoolingHttpClientConnectionManagerBuilder.create().setMaxConnTotal(400).setMaxConnPerRoute(20).build();
}

@Bean
public ClientHttpRequestFactory gatewayHttpClientFactory(final PoolingHttpClientConnectionManager poolingHttpClientConnectionManager) {
    // configure http client used for API gateway calls
    CloseableHttpClient httpClient = HttpClients
        .custom()
        .setConnectionManager(poolingHttpClientConnectionManager)
        .evictExpiredConnections()
        .evictIdleConnections(TimeValue.of(2L, TimeUnit.MINUTES))
        .build();
    return new HttpComponentsClientHttpRequestFactory(httpClient);
}

Interestingly, this changed the pool for the gateway routes, but not for the Eureka server communication. That must use another HTTP connection pool with the defaults (5/25).