We are developing/supporting a mobile app for a client that uses React-Native talking to a GCP backend. The database is Firestore, object storage uses GCP Storage, we use GCP app-engine for a few REST calls, and a series of Cloud Functions to handle some Firestore and GCP Storage processing.
Our prod deploys consist of delivering the mobile-app build files to the client (they distribute app internally on their own servers) and for the backend we use GCP CloudBuild to deploy our app-engine and cloud functions from our master/prod branch. However we have had a series of rough prod deploys over the last six months that largely boil down to unexpected environment config issues when we didn't encounter on dev or qa envs (or forgot about), as well as a few db update issues (datamodel changes). This leaves us with egg on our face with the client as the rollout can be delayed by a number of hours as we fix the issues.
To try and resolve this we have been considering a blue-green deployment strategy for the GCP backend. This would consist of creating a new GCP project, deploying latest master/prod branch to it, and allowing us to test fully before delivering updated mobile app to client - no surprises. This sounds great in theory but in practice it's not so simple. The main issue is the Firestore database and GCP Storage - we would also need to transfer over the storage objects and Firestore database to new project. Doable, but yet another complication.
So our Blue-Green idea is this:
We are hoping that pattern would allow us to do a Blue-Green backend logic deployment without needing to also worry about data/object transfers. My question is this: has anyone ever used this type of deployment pattern in GCP? Is it a recommended pattern or even a shunned anti-pattern?
Any feedback is appreciated. Thanks
I agree that something crucial like a mobile backend must be tested before the launch.
Accessing to the APIs of the Virtual Machines using gcloud, or migrating the traffic accordingly to the approve of the new updates, is something familiar to Cloud Build using the builders provided.
https://cloud.google.com/cloud-build/docs/cloud-builders#supported_builder_images_provided_by
Cloud Build lets you to deploy according to a conceptual staging or phasing that you could define previously (e.g. blue/green strategy). During the definition, you have to keep in mind the type of application to deploy; in this case you have mentioned a Firestore instance and the general suggestion is to replicate in another instance to get all arranged.
https://cloud.google.com/appengine/docs/flexible/nodejs/migrating-traffic
In the other hand, you can have a kubernetes cluster that allows you to manage these features in a more natural way, so maybe you have to evaluate the workload of your application and try Cloud build beyond the patterns.
I hope this point of view was useful for you.