I am using Consul-Template to render a properties with the data gathered from Consul. The files are being rendered successfully but afterwards Consul-Template fails over and over again until it crashes:
2017/08/16 10:43:04.900637 [WARN] (view) kv.list(...): Unexpected response code: 504 (retry attempt 5 after "4s")
This goes on and on with longer pauses until limit is reached / it crashes.
Configuration pulled from the logs:
2017/08/16 10:38:02.403969 [DEBUG] (runner) final config:
{
"Consul": {
"Address": "consul-client.development.discovery",
"Auth": {
"Enabled": false,
"Username": "",
"Password": ""
},
"Retry": {
"Attempts": 12,
"Backoff": 250000000,
"MaxBackoff": 60000000000,
"Enabled": true
},
"SSL": {
"CaCert": "",
"CaPath": "",
"Cert": "",
"Enabled": false,
"Key": "",
"ServerName": "",
"Verify": true
},
"Token": "",
"Transport": {
"DialKeepAlive": 30000000000,
"DialTimeout": 30000000000,
"DisableKeepAlives": false,
"IdleConnTimeout": 90000000000,
"MaxIdleConns": 100,
"MaxIdleConnsPerHost": 3,
"TLSHandshakeTimeout": 10000000000
}
},
"Dedup": {
"Enabled": false,
"MaxStale": 2000000000,
"Prefix": "consul-template/dedup/",
"TTL": 15000000000
},
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 0
},
"KillSignal": 2,
"LogLevel": "DEBUG",
"MaxStale": 2000000000,
"PidFile": "",
"ReloadSignal": 1,
"Syslog": {
"Enabled": false,
"Facility": "LOCAL0"
},
"Templates": [
{
"Backup": false,
"Command": "",
"CommandTimeout": 30000000000,
"Contents": "",
"Destination": "/consul-template/output/service.properties",
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 30000000000
},
"Perms": 420,
"Source": "/consul-template/data/service.properties.tpl",
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
},
"LeftDelim": "",
"RightDelim": ""
},
{
"Backup": false,
"Command": "",
"CommandTimeout": 30000000000,
"Contents": "",
"Destination": "/consul-template/output/application.properties",
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 30000000000
},
"Perms": 420,
"Source": "/consul-template/data/application.properties.tpl",
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
},
"LeftDelim": "",
"RightDelim": ""
},
{
"Backup": false,
"Command": "",
"CommandTimeout": 30000000000,
"Contents": "",
"Destination": "/consul-template/output/anotherMicroConsul.properties",
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 30000000000
},
"Perms": 420,
"Source": "/consul-template/data/anotherMicroConsul.properties.tpl",
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
},
"LeftDelim": "",
"RightDelim": ""
},
{
"Backup": false,
"Command": "",
"CommandTimeout": 30000000000,
"Contents": "",
"Destination": "/consul-template/output/master.properties",
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 30000000000
},
"Perms": 420,
"Source": "/consul-template/data/master.properties.tpl",
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
},
"LeftDelim": "",
"RightDelim": ""
},
{
"Backup": false,
"Command": "",
"CommandTimeout": 30000000000,
"Contents": "",
"Destination": "/consul-template/output/microConsul.properties",
"Exec": {
"Command": "",
"Enabled": false,
"Env": {
"Blacklist": [],
"Custom": [],
"Pristine": false,
"Whitelist": []
},
"KillSignal": 2,
"KillTimeout": 30000000000,
"ReloadSignal": null,
"Splay": 0,
"Timeout": 30000000000
},
"Perms": 420,
"Source": "/consul-template/data/microConsul.properties.tpl",
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
},
"LeftDelim": "",
"RightDelim": ""
}
],
"Vault": {
"Address": "",
"Enabled": false,
"Grace": 15000000000,
"RenewToken": true,
"Retry": {
"Attempts": 12,
"Backoff": 250000000,
"MaxBackoff": 60000000000,
"Enabled": true
},
"SSL": {
"CaCert": "",
"CaPath": "",
"Cert": "",
"Enabled": true,
"Key": "",
"ServerName": "",
"Verify": true
},
"Transport": {
"DialKeepAlive": 30000000000,
"DialTimeout": 30000000000,
"DisableKeepAlives": false,
"IdleConnTimeout": 90000000000,
"MaxIdleConns": 100,
"MaxIdleConnsPerHost": 3,
"TLSHandshakeTimeout": 10000000000
},
"UnwrapToken": false
},
"Wait": {
"Enabled": false,
"Min": 0,
"Max": 0
}
}
Sifting through the code this error message seems to be coming from Renew in session.go.
Am I supposed to see a session in Consul under /session/list? I have multiple Consul-Template containers running (one on each host) - Am I supposed to see several sessions? Right now I don't see anything...
curl localhost:8500/v1/session/list
[]
Cause: You have a proxy or a load balancer between your consul-template and consul.
Workaround: Increase your load balancer or proxy timeout for idle connections to something above 60sec (e.g: 65sec or 120sec) because after that keepalive will kick in and will keep the connection alive :)
Details