upstream connect error or disconnect/reset before headers. reset reason: connection termination when using Spring Boot

I am using Spring Boot with Embedded Tomcat 9.0.36. It is used as a Docker image in Kubernetes. Recently after upgrading envoy, I started getting exceptions.

  "upstream connect error or disconnect/reset before headers. reset reason: connection termination" with 503 status code

Some people suggested increasing idle connection Time out to 60 seconds but it spring-boot I was able to find out "Connection Time Out" & "Keep-Alive Time Out". I increased them to 5 minutes using the below code.

@Configuration
public class TomcatCustomizer implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {

    private static final Logger LOGGER = LoggerFactory.getLogger(TomcatCustomizer.class);

    @Override
    public void customize(TomcatServletWebServerFactory factory) {

        factory.addConnectorCustomizers(connector -> {
            AbstractHttp11Protocol protocol = (AbstractHttp11Protocol) connector.getProtocolHandler();
            //Setting up connection time out
            protocol.setKeepAliveTimeout(360000);
            protocol.setConnectionTimeout(360000);
            protocol.setMaxKeepAliveRequests(120);
        });
    }
}

Still, I am getting the same error. This application calls another service internally which is also hosted in Kubernetes. I am able to see a successful response in my service but after that, I don't see any logs.

Solution

I spent a week analyzing this from the application point of view. I followed a few steps that were suggested by the Ops Team.

Increase Timeout in Tomcat Server to 60 seconds because they have configured the same in Envoy
I did increase the time but not able to solve the issue.
I was using Spring Cloud Gateway for Gateway service I thought that is the issue so I changed it into Rest Templates but that also didn't solve the issue.
Luckily Health Check APIs working fine except those which were communicating to other services internally. In Health APIs they were also communicating with other services to check their Health but I was not returning response directly. I was wrapping up the response body modifying it and den forwarding it to UI. I also applied the same and use the below code which you can understand easily.I created a new Response Entity and dropped all headers which I received from internal APIs and returned to UI. It worked like charm.

//Earlier (Forwarding same headers received from internal service to UI)
ResponseEntity responseEntity = //Received by calling other APIs;
return responseEntity;

//Now (Dropped headers)
ResponseEntity responseEntity = //Received by calling other APIs;
MultiValueMap<String, String> newHeaders = new LinkedMultiValueMap<>();          
if (Objects.nonNull(responseEntity) && Objects.nonNull(responseEntity.getBody())) {
    newHeaders.set("Content-type", responseEntity.getHeaders().getContentType().toString());
    return new ResponseEntity(responseEntity.getBody(), newHeaders, responseEntity.getStatusCode());
}