[SOLVED] etcd backup to internal S3

etcd backup to internal S3

I'm trying to backup my rke cluster to my own internal hosted minio server. But I keep hitting this error.

FATA[0002] Failed to take one-time snapshot, exit code [1]: time="2019-04-29T08:37:15Z" level=fatal msg="faield to set s3 server: failed to check s3 bucket:rke, err:Get https://redacted/rke/?location=: x509: certificate signed by unknown authority"

I'm running Rancher v2.3.2 and rke v0.3.2 with Docker 17.03.05

Solution

The issue is caused because the container doesn't trust the certificate provided by the S3 server. This is mostly because it's signed by an internal CA. To resolve this issue, please added the field custom_ca to your cluster.yaml with your root ca cert.

Example:

services:
  etcd:
    backup_config:
      interval_hours: 12
      retention: 6
      s3backupconfig:
        access_key: S3_ACCESS_KEY
        secret_key: S3_SECRET_KEY
        bucket_name: s3-bucket-name
        region: ""
        endpoint: s3.rancher.support
        custom_ca: "-----BEGIN CERTIFICATE-----\nMIIDazCCAlOgAwIBAgIUMo....\n-----END CERTIFICATE-----"

You can find the full doc here https://support.rancher.com/hc/en-us/articles/360033950632-Is-it-possible-to-perform-etcd-snapshots-to-an-s3-endpoint-with-a-certificate-signed-by-a-custom-CA-