spring-boottomcatkubernetesistio

upstream connect error or disconnect/reset before headers. reset reason: connection termination when using Spring Boot


I am using Spring Boot with Embedded Tomcat 9.0.36. It is used as a Docker image in Kubernetes. Recently after upgrading envoy, I started getting exceptions.

  "upstream connect error or disconnect/reset before headers. reset reason: connection termination" with 503 status code

Some people suggested increasing idle connection Time out to 60 seconds but it spring-boot I was able to find out "Connection Time Out" & "Keep-Alive Time Out". I increased them to 5 minutes using the below code.

@Configuration
public class TomcatCustomizer implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    private static final Logger LOGGER = LoggerFactory.getLogger(TomcatCustomizer.class);

    @Override
    public void customize(TomcatServletWebServerFactory factory) {

        factory.addConnectorCustomizers(connector -> {
            AbstractHttp11Protocol protocol = (AbstractHttp11Protocol) connector.getProtocolHandler();
            //Setting up connection time out
            protocol.setKeepAliveTimeout(360000);
            protocol.setConnectionTimeout(360000);
            protocol.setMaxKeepAliveRequests(120);
        });
    }
}

Still, I am getting the same error. This application calls another service internally which is also hosted in Kubernetes. I am able to see a successful response in my service but after that, I don't see any logs.


Solution

  • I spent a week analyzing this from the application point of view. I followed a few steps that were suggested by the Ops Team.

    //Earlier (Forwarding same headers received from internal service to UI)
    ResponseEntity responseEntity = //Received by calling other APIs;
    return responseEntity;
    
    //Now (Dropped headers)
    ResponseEntity responseEntity = //Received by calling other APIs;
    MultiValueMap<String, String> newHeaders = new LinkedMultiValueMap<>();          
    if (Objects.nonNull(responseEntity) && Objects.nonNull(responseEntity.getBody())) {
        newHeaders.set("Content-type", responseEntity.getHeaders().getContentType().toString());
        return new ResponseEntity(responseEntity.getBody(), newHeaders, responseEntity.getStatusCode());
    }