Docker container breaks localhost connection with CSF installed

I've been at this for 2 days now and can't wrap my head around it.

I have installed Docker on my server running WHM. Within Docker I have installed listmonk.app.

CONTAINER ID   IMAGE                      COMMAND                  CREATED        STATUS                   PORTS                                       NAMES
0eb46a14e3c2   listmonk/listmonk:latest   "./listmonk"             20 hours ago   Up 9 minutes             0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   listmonk_app
43b308157ebe   postgres:13-alpine         "docker-entrypoint.s…"   20 hours ago   Up 9 minutes (healthy)   0.0.0.0:9432->5432/tcp, :::9432->5432/tcp   listmonk_db

Next up, I installed a Cloudflare Tunnel to allow access via https://listmonk.ygprodeck.com to http://localhost:9000.

This works until I installed ConfigServer Security and Firewall (CSF) on the server. Port 9000 is blocked in this (not added via TCP_IN or TCP_OUT) which is what I would want but now my Cloudflare Tunnel cannot connect to the docker container as it times out.

On the server, I ran curl -v http://127.0.0.1:9000

* Rebuilt URL to: http://127.0.0.1:9000/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9000
> User-Agent: curl/7.61.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Disabling CSF fixes the issue, but then these ports are exposed, which is what I am trying to avoid.

Yesterday, Instead of using CloudFlare Tunnel, I used an Apache Reverse Proxy with basically the same end result.

Additionally, I attempted to add docker0 to ETH_DEVICE_SKIP which just opens port 9000 externally to the world (I believe this just causes it to skip the firewall rules altogether).

I found a Stack question that could possibly be similar but I am not overly familiar with Docker. With listmonk.app, I used the easy production install.

docker-compose.yml:

version: "3.7"

x-app-defaults: &app-defaults
  restart: unless-stopped
  image: listmonk/listmonk:latest
  ports:
    - "9000:9000"
  networks:
    - listmonk
  environment:
    - TZ=Etc/UTC

x-db-defaults: &db-defaults
  image: postgres:13-alpine
  ports:
  networks:
    - listmonk
  environment:
    - POSTGRES_PASSWORD=<REMOVED>
    - POSTGRES_USER=<REMOVED>
    - POSTGRES_DB=<REMOVED>
  restart: unless-stopped
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U listmonk"]  
    interval: 10s
    timeout: 5s
    retries: 6

services:
  db:
    <<: *db-defaults
    container_name: listmonk_db
    volumes:
      - type: volume
        source: listmonk-data
        target: /var/lib/postgresql/data

  app:
    <<: *app-defaults
    container_name: listmonk_app
    depends_on:
      - db
    volumes:
      - ./config.toml:/listmonk/config.toml
      
  demo-db:
    container_name: listmonk_demo_db
    <<: *db-defaults

  demo-app:
    <<: *app-defaults
    container_name: listmonk_demo_app
    command: [sh, -c, "yes | ./listmonk --install --config config-demo.toml && ./listmonk --config config-demo.toml"]
    depends_on:
      - demo-db

networks:
  listmonk:

volumes:
  listmonk-data:

I will also include the listmonk.app config.toml file but I think that mostly needs to be kept as is:

[app]
# Interface and port where the app will run its webserver.  The default value
# of localhost will only listen to connections from the current machine. To
# listen on all interfaces use '0.0.0.0'. To listen on the default web address
# port, use port 80 (this will require running with elevated permissions).
address = "0.0.0.0:9000"

# BasicAuth authentication for the admin dashboard. This will eventually
# be replaced with a better multi-user, role-based authentication system.
# IMPORTANT: Leave both values empty to disable authentication on admin
# only where an external authentication is already setup.
admin_username = "<REMOVED>"
admin_password = "<REMOVED>"

# Database.
[db]
host = "listmonk_db"
port = 5432
user = "<REMOVED>"
password = "<REMOVED>"

# Ensure that this database has been created in Postgres.
database = "listmonk"

ssl_mode = "disable"
max_open = 25
max_idle = 25
max_lifetime = "300s"

# Optional space separated Postgres DSN params. eg: "application_name=listmonk gssencmode=disable"
params = ""

Solution

Docker and Cloudflare Tunnel client

You did not mention how you deployed the Cloudflare Tunnel client. As you already deployed listmonk via Docker, it only makes sense to do the same with the CF Tunnel client and use a shared Docker network for both services.

With the following compose file, you add a CF Tunnel client to the existing docker network listmonk:

version: '3'

networks:
  listmonk:
    external: true

services:
  cloudflaretunnel:
    image: cloudflare/cloudflared
    environment:
      - TUNNEL_TOKEN=$TUNNEL_TOKEN
    command: tunnel --no-autoupdate run
    restart: unless-stopped
    networks:
      - listmonk

The benefit is you don't need to expose listmonk service on 0.0.0.0:9000 where your firewall can interfere. Also, 0.0.0.0 meaning you're allowing remote connections. So you can get rid of:

  ports:
    - "9000:9000"

in the listmonk compose file.

(link to cloudflared Docker image on DockerHub)

What about Cloudflare Tunnel client and a Firewall?

From the documentation:

You can implement a positive security model with Cloudflare Tunnel by blocking all ingress traffic and allowing only egress traffic from cloudflared. Only the services specified in your tunnel configuration will be exposed to the outside world.

You only need to allow outbound connection on port 7844:

cloudflared connects to Cloudflare’s global network on port 7844. To use Cloudflare Tunnel, your firewall must allow outbound connections to the following destinations on port 7844 (via UDP if using the quic protocol or TCP if using the http2 protocol).