nginxtimeoutnginx-reverse-proxykongrequest-timed-out

How to set timeout in Kong v1.1.2

Problem

I am getting an error message in my Kong error log reporting that the upstream server has timed out. But I know that the upstream process was just taking over a minute, and when it completes (after Kong has logged the error) it logs a java error "Broken Pipe", implying that Kong was no longer listening for the response.

This is the behavior when the upstream process takes longer than 60 seconds. In some cases, it takes less than 60 seconds and everything works correctly.

How can I extend Kong's timeout?

Details

Kong Version

1.1.2

Kong's Error Message (slightly edited):

2019/12/06 09:57:10 [error] 1421#0: *1377 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xyz.xyz.xyz.xyz, server: kong, request: "POST /api/...... HTTP/1.1", upstream: "http://127.0.0.1:8010/api/.....", host: "xyz.xyz.com"

Here is the error from the upstream server log (Java / Tomcat via SpringBoot)

Dec 06 09:57:23 gateway-gw001-99 java[319]: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
Dec 06 09:57:23 gateway-gw001-99 java[319]:         at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:364) ~[tomcat-embed-core-8.5.42.jar!/
Dec 06 09:57:23 gateway-gw001-99 java[319]:         at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:833) ~[tomcat-embed-core-8.5.42.jar!
...

My kong.conf (slightly edited)

trusted_ips = 0.0.0.0/0
admin_listen = 0.0.0.0:8001
proxy_listen = 0.0.0.0:8080 proxy_protocol,  0.0.0.0:8443 ssl proxy_protocol
database = postgres
pg_host = 127.0.0.1
pg_port = 5432
pg_user = kong
pg_password = xyzxyzxyzxyzxyz
pg_database = kong
plugins = bundled,session
real_ip_header = proxy_protocol

A little more Context

The Outer NGinX Stream Config:

stream {

    server {
        listen 80;
        proxy_pass xyz.xyz.xyz.xyz:8080;
        proxy_protocol on;
    }

    server {
        listen 443;
        proxy_pass xyz.xyz.xyz.xyz:8443;
        proxy_protocol on;
    }
}

What I've Tried

I've tried adding the following lines to kong.conf. In version 1.1.2 of Kong you basically alter the NGinX settings remotely by adding prefixes to NginX config and placing them in the kong.conf (https://docs.konghq.com/1.1.x/configuration/#injecting-individual-nginx-directives ). None of them seemed to do anything:

nginx_http_keepalive_timeout=300s
nginx_proxy_proxy_read_timeout=300s
nginx_http_proxy_read_timeout=300s
nginx_proxy_send_timeout=300s
nginx_http_send_timeout=300s
Solution

  • Per the documentation Kong Version 0.10 has three properties that you can set for managing proxy connections

    1. upstream_connect_timeout: defines in milliseconds the timeout for establishing a connection to your upstream service.
    2. upstream_send_timeout: defines in milliseconds a timeout between two successive write operations for transmitting a request to your upstream service.
    3. upstream_read_timeout: defines in milliseconds a timeout between two successive read operations for receiving a request from your upstream service.

    In this case, as Kong is timing out waiting for the response from the upstream you would need to add a property setting for upstream_read_timeout

    In the Kong Version 1.1 documentation the Service object now includes these timeout attributes with slightly different names:

    1. connect_timeout: The timeout in milliseconds for establishing a connection to the upstream server. Defaults to 60000.
    2. write_timeout: The timeout in milliseconds between two successive write operations for transmitting a request to the upstream server. Defaults to 60000.
    3. read_timeout: The timeout in milliseconds between two successive read operations for transmitting a request to the upstream server. Defaults to 60000.