I am deploying a Spring Boot app to an Azure App Service. This app connects to a database that is configured to use flyway migrations.
When trying to release my latest version I have a flyway migration script that is taking a very long time to complete. This is expected as I am working with a very large DB and the migration script is running some hefty queries.
Unfortunately I am unable to complete this flyway migration as the app will automatically restart after 5 minutes, crashing the migration process and severing the connection to the DB. This happens since it considers the startup sequence to have failed whilst the migration is still ongoing.
Error I am getting in the deployment logs:
Starting container for site
INFO - docker run (... my site info)
ERROR - Container xxx for site yyy did not start within expected time limit. Elapsed time = 230.8677441 sec
I believe this is expected behavior as the deployment process is waiting for a successful health check to be returned from the app before considering it is up. Whilst the migration is ongoing, the app is not up and not responding to any health checks.
I tried setting the WEBSITES_CONTAINER_START_TIME_LIMIT env variable to the maximum allowed value of 1800 Sec (30 minutes). But this isn't enough time for my migration to completely finish and the migration process is killed after 30 mins failing the whole deployment.
How should I proceed to allocate more time for the app to execute the migration before Azure kills the app?
Many thanks
Decouple the Flyway migrations from the app startup. add a step in the CI/CD pipeline to run Flyway migrations using the same image before deploying the app to Azure Web Service.
pipeline step for Azure DevOps:
- task: Bash@3
displayName: "Run Flyway migrations"
inputs:
targetType: "inline"
script: |
docker run --rm \
-e FLYWAY_URL=jdbc:<db-url> \
-e FLYWAY_USER=<db-user> \
-e FLYWAY_PASSWORD=<db-password> \
<acr-name>.azurecr.io/<image-name>:<tag> \
flyway migrate
Tag the image specifically for migration purposes, allowing you to deploy the app without running the migrations during startup.
Add a command in Dockerfile:
CMD ["java", "-Dspring.profiles.active=migration", "-jar", "app.jar"]
During CI/CD, pull and run the image with the migration profile before deploying it to Azure.
docker run -e SPRING_PROFILES_ACTIVE=migration <acr-name>.azurecr.io/<image-name>:<tag>
200 OK during migrations. For example, use a lightweight response like "migration in progress".@RestController
public class HealthCheckController {
@GetMapping("/health")
public ResponseEntity<String> health() {
if (migrationInProgress) {
return ResponseEntity.ok("Migration in progress");
}
return ResponseEntity.ok("Healthy");
}
}
If the CI/CD pipeline can't accommodate migrations due to time constraints, run them in an Azure Container Instance (ACI)
az container create \
--resource-group <resource-group> \
--name db-migration \
--image <acr-name>.azurecr.io/<image-name>:<tag> \
--registry-login-server <acr-name>.azurecr.io \
--registry-username <acr-username> \
--registry-password <acr-password> \
--command-line "java -Dspring.profiles.active=migration -jar app.jar"
Pipeline log build stage:
2024-11-20 10:00:00 Starting job: Build
2024-11-20 10:00:01 Checking out repository...
> actions/checkout@v4
ā Checked out repository code
2024-11-20 10:00:10 Setting up Java version...
> actions/setup-java@v1
ā Set Java version to 17
2024-11-20 10:00:20 Building project with Maven...
$ mvn clean install
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------------< com.example:my-app >------------------------
[INFO] Building my-app 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ my-app ---
[INFO] Deleting /home/runner/work/my-app/target
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ my-app ---
[INFO] Copying resources to /home/runner/work/my-app/target/classes
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ my-app ---
[INFO] Compiling 20 source files to /home/runner/work/my-app/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ my-app ---
[INFO] Copying resources to /home/runner/work/my-app/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ my-app ---
[INFO] Tests are passing
[INFO]
[INFO] --- maven-jar-plugin:3.1.0:jar (default-jar) @ my-app ---
[INFO] Building jar: /home/runner/work/my-app/target/my-app-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ my-app ---
[INFO] Installing /home/runner/work/my-app/target/my-app-1.0-SNAPSHOT.jar to local repository
[INFO] Build success
2024-11-20 10:05:00 Build completed successfully.
flyway Migration:
2024-11-20 10:05:10 Starting job: Migrations
2024-11-20 10:05:11 Pulling Docker image from ACR...
$ docker pull myacr.azurecr.io/my-app:latest
latest: Pulling from myacr.azurecr.io/my-app
Digest: sha256:abcdef1234567890
Status: Image is up to date for myacr.azurecr.io/my-app:latest
2024-11-20 10:05:30 Running Flyway migrations...
$ docker run --rm \
-e FLYWAY_URL=jdbc:postgresql://mydb.postgres.database.azure.com:5432/mydatabase \
-e FLYWAY_USER=myuser \
-e FLYWAY_PASSWORD=mypassword \
myacr.azurecr.io/my-app:latest \
flyway migrate
2024-11-20 10:05:31 INFO - Starting Flyway Command-Line Tool
2024-11-20 10:05:31 INFO - Database: jdbc:postgresql://mydb.postgres.database.azure.com:5432/mydatabase (PostgreSQL 12)
2024-11-20 10:05:31 INFO - Flyway Community Edition 9.0.0 by Redgate
2024-11-20 10:05:31 INFO - Validating migrations ...
2024-11-20 10:05:32 INFO - Successfully validated 5 migrations (execution time 00:01.001s)
2024-11-20 10:05:32 INFO - Current version of schema `public`: 2.3
2024-11-20 10:05:32 INFO - Migrating schema `public` to version 2.4 - add new indexes
2024-11-20 10:15:40 INFO - Successfully applied migration to version 2.4 (execution time 00:10:08.123s)
2024-11-20 10:15:40 INFO - Migrating schema `public` to version 2.5 - update large table
2024-11-20 10:45:45 INFO - Successfully applied migration to version 2.5 (execution time 00:30:04.891s)
2024-11-20 10:45:45 INFO - Successfully applied 2 migrations (execution time 00:40:15.015s)
2024-11-20 10:45:45 INFO - Schema `public` is now at version 2.5
2024-11-20 10:45:45 INFO - Migration completed successfully.
2024-11-20 10:46:00 Flyway migrations completed.
Deploy Application:
2024-11-20 10:46:10 Starting job: Deploy
2024-11-20 10:46:11 Downloading artifact from build job...
> actions/download-artifact@v3
Artifact 'java-app' downloaded successfully.
2024-11-20 10:46:20 Deploying to Azure Web App...
> azure/webapps-deploy@v2
[INFO] Using publish profile from secret.
[INFO] Deploying jar file: /home/runner/work/my-app/target/my-app-1.0-SNAPSHOT.jar to app 'sp-az-webapp'
2024-11-20 10:47:00 INFO - Pulling image from ACR: myacr.azurecr.io/my-app:latest
2024-11-20 10:47:20 INFO - Image pulled successfully
2024-11-20 10:47:25 INFO - Deploying container to Azure Web App: sp-az-webapp
2024-11-20 10:47:30 INFO - Starting container for site
2024-11-20 10:47:32 INFO - Initiating container start-up
2024-11-20 10:47:40 INFO - Waiting for site to become ready ...
2024-11-20 10:48:00 INFO - Site is up and running
2024-11-20 10:48:00 INFO - Deployment completed successfully.
Web App URL: https://sp-az-webapp.azurewebsites.net