Distributed secure MinIO in docker-compose
I have a quite complex system in Docker. Everything runs in a big docker-compose file. Previously everything runs on one (manager) node in my Docker Swarm so I have generated a CERT for my domain (with certbot) and I have used the below MinIO service in my compose file:
object_storage:
image: minio/minio:RELEASE.2020-12-10T01-54-29Z
ports:
- 9000:9000
environment:
MINIO_ACCESS_KEY_FILE: object_storage_user
MINIO_SECRET_KEY_FILE: object_storage_password
command: server /data
depends_on:
- fluentd
volumes:
- object_storage_data:/data
- ./certs/domain.crt:/root/.minio/certs/public.crt
- ./certs/domain.key:/root/.minio/certs/private.key
networks:
- object_storage_net
secrets:
- object_storage_user
- object_storage_password
logging:
driver: "fluentd"
options:
fluentd-address: ${SYSTEM_HOST}:24224
tag: object-storage
The above implementation works as expected! But now I have 2 separated servers to run the MinIO. These servers are joined to my Docker Swarm as worker nodes. The MinIO shouldn't run on manager node (Only on two separated worker nodes)!
>>> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
mcbkz9m5nzf7oa3fiqk0lf4qo * manager Ready Active Leader 20.10.1
dz4e3k70g8ik2z4bcx8u0ft9ao minio_1 Ready Active 20.10.2
r0qpdn2guyy5773vo8vg2trzo minio_2 Ready Active 20.10.2
My current MinIO implementation in my docker-compose file:
object_storage_1:
image: minio/minio:RELEASE.2020-12-10T01-54-29Z
ports:
- 9000:9000
environment:
MINIO_ACCESS_KEY_FILE: object_storage_user
MINIO_SECRET_KEY_FILE: object_storage_password
command: server https://object_storage_{1...2}/data{1...2}
depends_on:
- fluentd
volumes:
- object_storage_data_1_1:/data1
- object_storage_data_1_2:/data2
- ./certs/domain.crt:/root/.minio/certs/public.crt
- ./certs/domain.key:/root/.minio/certs/private.key
networks:
- object_storage_net
secrets:
- object_storage_user
- object_storage_password
deploy:
restart_policy:
condition: on-failure
placement:
constraints:
- node.hostname == minio_1
logging:
driver: "fluentd"
options:
fluentd-address: ${SYSTEM_HOST}:24224
tag: object-storage
object_storage_2:
image: minio/minio:RELEASE.2020-12-10T01-54-29Z
ports:
- 9000
environment:
MINIO_ACCESS_KEY_FILE: object_storage_user
MINIO_SECRET_KEY_FILE: object_storage_password
command: server https://object_storage_{1...2}/data{1...2}
depends_on:
- fluentd
volumes:
- object_storage_data_2_1:/data1
- object_storage_data_2_2:/data2
- ./certs/domain.crt:/root/.minio/certs/public.crt
- ./certs/domain.key:/root/.minio/certs/private.key
networks:
- object_storage_net
secrets:
- object_storage_user
- object_storage_password
deploy:
restart_policy:
condition: on-failure
placement:
constraints:
- node.hostname == minio_2
logging:
driver: "fluentd"
options:
fluentd-address: ${SYSTEM_HOST}:24224
tag: object-storage
If I check the log of an instance of my MinIO service, I got the following error:
Unable to read 'format.json' from https://object_storage_1:9000/data1: Post "https://object_storage_1:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_1
Unable to read 'format.json' from https://object_storage_2:9000/data1: Post "https://object_storage_2:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_2
But I can reach the MinIO on 9000 port, just there is a pop-up error:
I want to access to MinIO only through my domain (my_domain.app:9000). The MinIO in this case doesn't use the real server name however it uses the "virtual" Docker network (Eg.: https://object_storage_2:9000).
My questions:
- How can I generate certs for "virtual" Docker networks (Eg.: object_storage_1 or object_storage_2)?
- Where should I put the generated certs?
- Is is possible to solve with only my generate (for my domain) cert?
I am open for every hint and solution!
Solution
I had to put the (Domain) CERT file to minio/certs/CAs folder instead of /root/.minio/certs folder. Furthermore I had to copy the CERT to the worker nodes (separated servers) if I didn't copy it to nodes the service didn't find it on the worker node.
The correct volumes parameter looks like as below:
volumes:
- object_storage_data_1_1:/data1
- object_storage_data_1_2:/data2
- ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
One working service of my several MinIO services:
object-storage-1:
image: minio/minio:RELEASE.2021-08-17T20-53-08Z
expose:
- "9000"
- "9001"
environment:
MINIO_ACCESS_KEY_FILE: object_storage_user
MINIO_SECRET_KEY_FILE: object_storage_password
MINIO_BROWSER_REDIRECT_URL: https://${SYSTEM_HOST}:9001
MINIO_SERVER_URL: https://${SYSTEM_HOST}:9000
command: server --console-address ":9001" http://object-storage-{1...4}/data{1...2}
hostname: object-storage-1
depends_on:
- fluentd
volumes:
- object_storage_data_1_1:/data1
- object_storage_data_1_2:/data2
- ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
networks:
- object_storage_net
secrets:
- object_storage_user
- object_storage_password
deploy:
restart_policy:
condition: on-failure
placement:
constraints:
- node.hostname == minio1
logging:
driver: "fluentd"
options:
fluentd-address: ${SYSTEM_HOST}:24224
tag: object-storage
AND I had to create a NgInx config:
upstream minio {
server object-storage-1:9000;
server object-storage-2:9000;
server object-storage-3:9000;
server object-storage-4:9000;
}
upstream console {
ip_hash;
server object-storage-1:9001;
server object-storage-2:9001;
server object-storage-3:9001;
server object-storage-4:9001;
}
server {
listen 9000 ssl;
listen [::]:9000 ssl;
server_name my.server.com;
ssl_certificate /ssl/domain.crt;
ssl_certificate_key /ssl/domain.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001 ssl;
listen [::]:9001 ssl;
server_name my.server.com;
ssl_certificate /ssl/domain.crt;
ssl_certificate_key /ssl/domain.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
- Unable to pull images no such host on Windows
- Check if clickhouse is healthy docker-compose
- Docker Compose, Rails and Webpacker not perserving node_modules
- Need to use docker-compose to build but but not run
- Docker Error: Jupyter Notebook container is not running after successful build
- error during connect: This error may indicate that the docker daemon is not running
- Google Jib throws UnknownManifestFormatException: Unknown mediaType: application/vnd.oci.image.index.v1+json when building the application
- AWS ECS exec /usr/local/bin/docker-entrypoint.sh: exec format error
- Docker: Running nano in docker container
- Error building nextjs app on docker but not locally
- Tekton error , how to fix: Error executing command: fork/exec /tekton/scripts/script-2-m6dkb
- php_network_getaddresses: getaddrinfo failed error in Docker's adminer
- "update-locale: Error: invalid locale settings" when using a Dockerfile
- JVM -XX:MaxRAMPercentage not being applied in container
- docker-compose up not recreate container
- How can I find a Docker image with a specific tag in Docker registry on the Docker command line?
- Pass environment variables from docker-compose to container at build stage
- denied: requested access to the resource is denied when pushing image to gitlab registry
- How to use a devcontainer in vscode.dev?
- Unable to connect to mongodb database on docker container with nestJs
- Gitlab doesn't loads new ssl cert and key
- How to diagnose "docker: invalid reference format" when calling "docker run"?
- Azure Container Instance does not respond with dns name on browser
- Auto-generate OpenFGA json model when saving .fga DSL file
- KASM Stand Alone Service inaccessble behind Nginx: Websocket error or invalid HTTP version
- how to run NiFi-toolkit docker image
- How to check if a Docker image with a specific tag exist on registry?
- Can we mount sub-directories of a named volume in docker?
- AWS InvalidSignatureException, Signature expired when running from docker container
- No such host CI, docker