traefikdocker-registrynomad

Nomad + Traefik docker registry without tls (local testing only)

I have 2 machines in the same internal network. I installed nomad + traefik + docker registry into one of them at 192.168.1.48. Exposed docker registry trough direct host port at :9169 and trough Traefik at :38081. I don't want to setup tls yet, so I added both endpoints into docker daemon config.

{
  "insecure-registries": [
    "192.168.1.48:9169",
    "192.168.1.48:38081"
  ]
}

Checked both endpoints like this.

curl -v -I 192.168.1.48:38081/v2/
*   Trying 192.168.1.48:38081...
* Connected to 192.168.1.48 (192.168.1.48) port 38081
* using HTTP/1.x
> HEAD /v2/ HTTP/1.1
> Host: 192.168.1.48:38081
> User-Agent: curl/8.11.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 2
Content-Length: 2
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Date: Tue, 27 May 2025 13:22:02 GMT
Date: Tue, 27 May 2025 13:22:02 GMT
< Docker-Distribution-Api-Version: registry/2.0
Docker-Distribution-Api-Version: registry/2.0
<

* Connection #0 to host 192.168.1.48 left intact

curl -v -I 192.168.1.48:9169/v2/
*   Trying 192.168.1.48:9169...
* Connected to 192.168.1.48 (192.168.1.48) port 9169
* using HTTP/1.x
> HEAD /v2/ HTTP/1.1
> Host: 192.168.1.48:9169
> User-Agent: curl/8.11.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 2
Content-Length: 2
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
Docker-Distribution-Api-Version: registry/2.0
< Date: Tue, 27 May 2025 13:21:49 GMT
Date: Tue, 27 May 2025 13:21:49 GMT
<

* Connection #0 to host 192.168.1.48 left intact

curl -v -k -I https://192.168.1.48:38081/v2/
*   Trying 192.168.1.48:38081...
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* Connected to 192.168.1.48 (192.168.1.48) port 38081
* using HTTP/1.x
> HEAD /v2/ HTTP/1.1
> Host: 192.168.1.48:38081
> User-Agent: curl/8.11.0
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Date: Tue, 27 May 2025 17:46:15 GMT
Date: Tue, 27 May 2025 17:46:15 GMT
< Content-Length: 19
Content-Length: 19
<

* Connection #0 to host 192.168.1.48 left intact

curl -v -k -I https://192.168.1.48:9169/v2/
*   Trying 192.168.1.48:9169...
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - O token fornecido para a função é inválido
* closing connection #0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - O token fornecido para a função é inválido

But, only the direct port works...

docker push 192.168.1.48:9169/gestaoapi
Using default tag: latest
The push refers to repository [192.168.1.48:9169/gestaoapi]
f18232174bc9: Layer already exists
94dd5906d95c: Already exists
4f4fb700ef54: Layer already exists
f9825f5070d7: Layer already exists
6b82be40034d: Layer already exists
c00f5714e8aa: Layer already exists
4ed0b5d14f27: Layer already exists
latest: digest: sha256:9ef3eb60eec90e5a02378906c452e897d3e8a3a451a8063f26bcd589c601e1a6 size: 856

docker push 192.168.1.48:38081/gestaoapi
Using default tag: latest
The push refers to repository [192.168.1.48:38081/gestaoapi]
94dd5906d95c: Waiting
4f4fb700ef54: Waiting
f9825f5070d7: Waiting
f18232174bc9: Waiting
6b82be40034d: Waiting
4ed0b5d14f27: Waiting
c00f5714e8aa: Waiting
unknown: unexpected status from POST request to https://192.168.1.48:38081/v2/gestaoapi/blobs/uploads/: 404 Not Found

Nomad jobs for traefik and docker registry.

job "traefik" {
  datacenters = ["dc1"]
  type = "system"

  group "traefik" {

    task "traefik" {
      driver = "docker"

      config {
        image = "traefik:v3.4.0"
        network_mode = "host"
        volumes = [
          "local/traefik.yml:/etc/traefik/traefik.yml"
        ]
        args = ["--configFile=/etc/traefik/traefik.yml"]
      }

      template {
        data = <<EOF
entryPoints:
  web:
    address: ":30080"
#    forwardedHeaders:
#      insecure: true
  websecure:
    address: ":30443"
    http:
      tls: {}
#    forwardedHeaders:
#      insecure: true
  dockerregistry:
    address: ":38081"
    forwardedHeaders:
      insecure: true
  traefik:
    address: ":38080"

api:
  dashboard: true
  insecure: true

log:
  level: DEBUG

providers:
  nomad:
    endpoint:
      address: http://127.0.0.1:4646
      token: "${nomad_token}"
    exposedByDefault: false
    prefix: "traefik"
    refreshInterval: 15s
    watch: true
    namespaces: 
      - "default"
EOF
        destination = "local/traefik.yml"
      }
    }
  }
}

job "docker-registry" {
  datacenters = ["dc1"]
  type        = "service"

  update {
    max_parallel      = ${replica_count}
    canary            = ${replica_count}
    health_check      = "checks"
    min_healthy_time  = "30s"
    healthy_deadline  = "5m"
    progress_deadline = "10m"
    auto_revert       = true
    auto_promote      = true
    stagger           = "30s"
  }

  group "registry" {
    count = ${replica_count}

    network {
      mode = "bridge"  
      port "registry" {
        to = 5000
        static = 9169
      }
    }

    volume "registry-data" {
      type      = "host"
      source    = "docker-registry-data"
      read_only = false
    }

    service {
      name = "docker-registry"
      port = "registry"
      provider = "nomad"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.docker-registry-main.entrypoints=dockerregistry",
        "traefik.http.routers.docker-registry-main.rule=PathPrefix(`/`)"
      ]
      canary_tags = [
        "traefik.enable=true",
        "traefik.nomad.canary=true",
        "traefik.http.routers.docker-registry-canary.entrypoints=dockerregistry",
      ]

      check {
        type     = "http"
        path     = "/v2/"
        interval = "20s"
        timeout  = "2s"

        check_restart {
          limit = 3
          grace = "60s"
        }
      }
    }

    task "registry" {
      driver = "docker"

      config {
        image = "registry:2"
        ports = ["registry"]

        volumes = [
          "local/config.yml:/etc/docker/registry/config.yml"
        ]
      }

      template {
        data = <<EOF
version: 0.1
log:
  level: debug
  formatter: text
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
      enabled: true
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
http:
  addr: :5000
EOF
        destination = "local/config.yml"
      }

      volume_mount {
        volume      = "registry-data"
        destination = "/var/lib/registry"
        read_only   = false
      }
    }
  }
}

Traefik

for some reason Traefik is trying to redirect to https when called from docker push? I ran out of ideas, any tips?

Solution

  • Enabling sniStrict solves my problem.