consulconsul-template

Using consul-template to render 2000+ services to nginx.conf

We have an nginx conf file that is being produced with consul-template for 2000+ services, running Consul 1.6.1.

We're trying to upgrade to Consul 1.7.2 and running into an issue where consul-template DDoSes Consul node agent.

Node/client agent v1.6.1 works fine, but node/client agent v1.7.2 starts refusing connections from consul-template.

The template in question (simplified version) is:

{{- range services }}
{{- if (in .Tags "nginxqa1") }}
{{- range service .Name }}
{{ .Address }}
{{- end }}
{{- end }}
{{- end }}

Adding the following limits config entry on the node/client consul seems to fix the issue:

http_max_conns_per_client = 5000

But, are there any implications to this?

Is there a better way of doing this? I'm seeing these messages:

2020/05/01 18:24:51.653257 [WARN] (runner) watching 2631 dependencies - watching this many dependencies could DDoS your consul cluster

Any ideas / suggestions appreciated.

Solution

  • The http_max_conns_per_client parameter was introduced in hashicorp/consul#7159 to address CVE-2020-7219. The default value of 200 seemed to be a reasonable default for most use cases. However, there are scenarios such as yours where it makes sense to raise this value.

    The only downside to raising this is that any client which is able to communicate with the Consul API will be able to create up to 5000 connections. It is not possible to raise this limit only for a specific client. Keep that in mind as you push these changes to production.