I've set up the following Github action:
name: Fulfill Terraform Production
on:
workflow_dispatch:
push:
branches:
- main
paths:
- terraform/**
- packer/**
- src/**
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
jobs:
fulfill_pre_required_terraform:
name: Fulfill Pre-Required Terraform Production
runs-on: ubuntu-latest
outputs:
github-runner-label: ${{ steps.start-ec2-github-runner.outputs.label }}
github-runner-ec2-instance-id: ${{ steps.start-ec2-github-runner.outputs.ec2-instance-id }}
database-url: ${{ steps.terraform-pre-required-outputs.outputs.DATABASE_URL}}
env:
AWS_REGION: ${{ vars.AWS_REGION }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
TF_VAR_aws_region: ${{ vars.AWS_REGION }}
TF_VAR_smartabook_route53_zone_id: ${{ secrets.SMARTABOOK_ROUTE53_ZOND_ID }}
TF_VAR_smartabook_backend_ecr_repository_name: smartabook-backend
TF_VAR_smartabook_database_username: ${{ secrets.SMARTABOOK_DATABASE_USERNAME }}
TF_VAR_smartabook_database_password: ${{ secrets.SMARTABOOK_DATABASE_PASSWORD }}
TF_VAR_jwt_secret: ${{ secrets.JWT_SECRET }}
TF_VAR_salt: ${{ secrets.SALT }}
TF_VAR_sendgrid_api_key: ${{ secrets.SENDGRID_API_KEY }}
TF_VAR_sendgrid_from_email: ${{ secrets.SENDGRID_FROM_EMAIL }}
TF_VAR_sendgrid_from_name: ${{ secrets.SENDGRID_FROM_NAME }}
TF_VAR_admin_secret: ${{ secrets.ADMIN_SECRET }}
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ vars.AWS_REGION }}
- name: Check if GitHub runner AMI already exists
id: get-github-runner-ami-id
env:
GH_RUNNER_AMI_NAME: ${{ vars.GH_RUNNER_AMI_NAME}}
run: |
EXISTING_AMI_ID=$(aws ec2 describe-images \
--region ${{ vars.AWS_REGION }} \
--owners self \
--filters "Name=name,Values=$GH_RUNNER_AMI_NAME" \
--query 'Images[0].ImageId' \
--output text 2>/dev/null)
echo "Got AMI ID output: $EXISTING_AMI_ID"
echo "GH_RUNNER_AMI_ID=$EXISTING_AMI_ID" >> $GITHUB_OUTPUT
- name: Terraform setup
uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Terraform init
working-directory: terraform/prod
env:
TERRAFORM_REMOTE_BACKEND_S3_BUCKET_NAME: ${{ secrets.TERRAFORM_REMOTE_BACKEND_S3_BUCKET_NAME }}
run: |
terraform init \
-backend-config="bucket=$TERRAFORM_REMOTE_BACKEND_S3_BUCKET_NAME" \
-backend-config="region=$AWS_REGION"
- name: Terraform plan pre-required resources only
working-directory: terraform/prod
run: |
terraform plan -no-color -out pre_required.tfplan -target=module.backend_cluster.aws_ecr_repository.server -target=aws_db_instance.postgres
- name: Terraform apply pre-required resources only
working-directory: terraform/prod
run: |
terraform apply pre_required.tfplan
- name: Get Terraform pre-required outputs
id: terraform-pre-required-outputs
working-directory: terraform/prod
run: |
github_runner_subnet_id=$(terraform output -raw github_runner_subnet_id)
github_runner_security_group_id=$(terraform output -raw github_runner_security_group_id)
database_url=$(terraform output -raw database_url)
echo "GITHUB_RUNNER_SUBNET_ID=$github_runner_subnet_id" >> $GITHUB_OUTPUT
echo "GITHUB_RUNNER_SECURITY_GROUP_ID=$github_runner_security_group_id" >> $GITHUB_OUTPUT
echo "DATABASE_URL=$database_url" >> $GITHUB_OUTPUT
- name: Start EC2 GitHub runner
id: start-ec2-github-runner
uses: machulav/ec2-github-runner@v2
with:
mode: start
github-token: ${{ secrets.EC2_GITHUB_RUNNER_CREATION_GITHUB_TOKEN }}
ec2-image-id: ${{ steps.build-github-runner-ami.outputs.GH_RUNNER_AMI_ID || steps.get-github-runner-ami-id.outputs.GH_RUNNER_AMI_ID }}
ec2-instance-type: t3.xlarge
subnet-id: ${{ steps.terraform-pre-required-outputs.outputs.GITHUB_RUNNER_SUBNET_ID }}
security-group-id: ${{ steps.terraform-pre-required-outputs.outputs.GITHUB_RUNNER_SECURITY_GROUP_ID }}
fulfill_terraform:
name: Fulfill Terraform Production
needs: fulfill_pre_required_terraform
runs-on: ${{ needs.fulfill_pre_required_terraform.outputs.github-runner-label }}
continue-on-error: true
env:
AWS_REGION: ${{ vars.AWS_REGION }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
TF_VAR_aws_region: ${{ vars.AWS_REGION }}
TF_VAR_smartabook_route53_zone_id: ${{ secrets.SMARTABOOK_ROUTE53_ZOND_ID }}
TF_VAR_smartabook_backend_ecr_repository_name: smartabook-backend
TF_VAR_smartabook_database_username: ${{ secrets.SMARTABOOK_DATABASE_USERNAME }}
TF_VAR_smartabook_database_password: ${{ secrets.SMARTABOOK_DATABASE_PASSWORD }}
TF_VAR_jwt_secret: ${{ secrets.JWT_SECRET }}
TF_VAR_salt: ${{ secrets.SALT }}
TF_VAR_sendgrid_api_key: ${{ secrets.SENDGRID_API_KEY }}
TF_VAR_sendgrid_from_email: ${{ secrets.SENDGRID_FROM_EMAIL }}
TF_VAR_sendgrid_from_name: ${{ secrets.SENDGRID_FROM_NAME }}
TF_VAR_admin_secret: ${{ secrets.ADMIN_SECRET }}
DATABASE_URL: ${{ needs.fulfill_pre_required_terraform.outputs.database-url }}
steps:
- uses: actions/checkout@v4
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Get package.json version
id: package-version
run: echo "current_version=$(jq -r .version package.json)" >> $GITHUB_OUTPUT
- name: Build, tag, and push image of website to Amazon ECR
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
IMAGE_TAG: ${{ steps.package-version.outputs.current_version }}-${{ github.sha }}
run: |
docker build -t $ECR_REGISTRY/$TF_VAR_smartabook_backend_ecr_repository_name:$IMAGE_TAG -f ./Dockerfile --build-arg DATABASE_URL=$DATABASE_URL .
docker push $ECR_REGISTRY/$TF_VAR_smartabook_backend_ecr_repository_name:$IMAGE_TAG
In the "fulfill_terraform", when trying to use $DATABASE_URL (as you can see in step name: Build, tag, and push image of website to Amazon ECR), this environment variable value is empty. I can guarantee that in job fulfill_pre_required_terraform, in step name: Get Terraform pre-required outputs, this command: echo "DATABASE_URL=$database_url" >> $GITHUB_OUTPUT works. I checked it and printed (echo) the value of $database_url and the value was non-empty but actual database URL.
Note that the job fulfill_terraform runs on self hosted runner, in comparison to fulfill_pre_required_terraform which runs on ubuntu-latest of GitHub. This makes me think that this is the issue.
Is there a way to resolve this issue easily? I can only think this solution:
DATABASE_URL to AWS secret managerDATABASE_URL from AWS secret managerBut I'm trying to ensure there is no other easy fix for this..
The issue is that GitHub probably skipped your database-url output because it contains sensitive value. You simply need to add:
echo "::add-mask::$database_url"
Before your output to $GITHUB_OUTPUT command