I have a springboot web application deployed on cloud run and it's been running live on production for a few years.
cloud run instance url: https://myapp-prod-a7ow2sutca-de.a.run.app region: asia-east1
We also have a custom domain mapping: api.myapp.com -> https://myapp-prod-a7ow2sutca-de.a.run.app
Now we work with a payment gateway that will send us payment received notification via REST API and so we expose an endpoint api.myapp.com/payment-received that they will hit when a payment is received on the payment gateway's end.
Suddenly yesterday some notifications (intermittently) are missing and they told us that some of the requests sent out to api.myapp.com/payment-received were not getting any response. Further investigation led to api.myapp.com resolving to 74.125.68.121 which is not reachable (tried to ping it gives timeout). So requests coming out from one of their cluster always failed to reach us because api.myapp.com always resolved to this bad ip.
Then I created a new cloud run deployment api2.myapp.com in a different region and asked them to ping api2.myapp.com from their problematic cluster. And at first it resolves to a different ip address and it's pingable but after a few minutes it resolved back to 74.125.68.121 and not reachable again.
Other payment gateways are still working without a problem only this 1 payment gateway is giving this issue. They're claiming that it's Google Cloud's problem they are routing their requests to a bad IP.
We also have Cloudflare in front and they claimed that they have nothing to do with this, and confirmed that there's no configuration that will block certain IP's or anything weird.
What seems to be the problem here? Who should I contact in this case? It had been working for years just fine and suddenly this happened...
Also i am curious because Cloud Run is deployed on asia-east1 region why is it that when i ping api.myapp.com from my local terminal, i always see ip address originated from United States owned by Google? I was expecting to see ip address from country in asia-east1 region (Taiwan)
When a cloud run service is created it is automatically assigned with a dynamic IP which may change.
If a cloud run service connects to an external endpoint like database or API which is using an IP address-based firewall, you can configure a static IP then use it in service. Follow this official document for creating a static outbound ip.
If you need inbound public traffic to a static IP, you can try using a HTTP load balancer and serverless network endpoint group(NEG). For more information follow these official documents Serverless NEG and static outbound IP’s.
Although you say Cloudflare is not involved, it's worth checking with them to ensure there are no configurations or edge caching rules that might be causing a redirect or DNS resolution issues.
IP addresses originated from the United States, this is because they belong to the same global IP block that Google has.
There might be a DNS issue, try requesting the payment gateway to clear your DNS cache and retry. Still, if you want to know about your current configuration then you can create a new Issue Tracker thread describing your issue. If you have paid support try create an issue.