I've a Global external Application Load Balancer on GCP serving several domains. As backends, I've VMs, buckets and some Cloud Run services, that will eventually replace VMs.
Everything seems to be working fine, I have some domains correctly served from the VMs and some from Cloud Run. I'm now trying to set up traffic mirroring to test development branches, but I'm unable to make it work. I'd like to mirror production traffic to a Cloud Run service that is currently working as primary service for a dedicated host/domain. To recap: the CR service is managing traffic for training.appname.domainname.ee
.
My backends are listening to HTTP (no HTTPS), as TLS is terminated at Load Balancer. The backends are not directly exposed to client traffic.
This is the LB rule associated to the working training.appname.domainname.ee
:
defaultService: projects/app01-1234/global/backendServices/cloudrun-app01-web-training
name: matcher-training-es
And it's serving greatly. Now, for a production website where I would like to get the traffic from, I've configured the following:
defaultService: projects/app01-1234/global/backendServices/lb01-backend-spain
name: path-matcher-5
routeRules:
- matchRules:
- prefixMatch: /
priority: 10
routeAction:
weightedBackendServices:
- backendService: projects/app01-1234/global/backendServices/lb01-backend-spain
headerAction:
requestHeadersToAdd:
- headerValue: 'true'
headerName: X-Is-Mirrored
replace: true
weight: 100
requestMirrorPolicy:
backendService: projects/app01-1234/global/backendServices/cloudrun-app01-web-training
This configuration is NOT working. It correctly delivers traffic to the primary service, but not to the mirrored one.
However, if I use a VM backend as requestMirrorPolicy.backendService
I correctly receive the traffic in access.log
(and via tcpdump
).
I initially had problems because I split traffic on the CR service via Apache2 vhosts, and I later found out the Load Balancer will add a suffix to the Host
header. But even handling this case, Cloud Run isn't (apparently) receiving mirrored requests.
To overcome this latest case, I also added to the CR service a ServerName _default_
virtualhost, but I'm not receiving anything yet.
I'm also struggling with logging. For what I found, there's no logging for the Load Balancer itself, only for the backends. I enabled logging on the Cloud Run backend, and while it works when visiting the dedicated host (training
), no record is printed when it should be hit by the mirrored traffic.
I'm out of ideas for debugging, so any help would be appreciated.
It seems I didn't RTFM:
Traffic mirroring is supported when both backend services have managed instance groups, zonal NEGs, or hybrid NEGs backends. It is not supported for internet NEGs, serverless NEGs, and Private Service Connect backends.
So Cloud Run cannot be used as a destination for traffic mirroring.