Cloud Run: 429: The request was aborted because there was no available instance

We (as a company) experience large spikes every day. We use Pub/Sub -> Cloud Run combination.

The issue we experience is that when high traffic hits, Pub/Sub tries to push messages to Cloud/Run all at the same time without any flow control. The result?

429: The request was aborted because there was no available instance.

Although this is marked as a warning, every 4xx HTTP response results in the message retry delivery.

Messages, therefore, come back to the queue and wait. If a message repeats this process and the instances are still taken, Cloud Run returns 429 again, and the message is sent back to the queue. This process repeats x times (depends on what value we set in Maximum delivery attempts). After that, the message goes to the dead-letter queue.

We want to avoid this and ideally don't get any 429, so the message won't travel back and forth, and it won't end up in the dead-letter subscription because it is not one of the application errors we want to keep there, but rather a warning caused by Pub/Sub not controlling the flow and coordinating with Cloud Run.

Neither Pub/Sub nor a push subscription (which is required to use for Cloud Run) have any flow control feature.

Is there any way to control how many messages are sent to Cloud Run to avoid getting the 429 response? And also, why does Pub/Sub even try to deliver when it is obvious that Cloud Run hit the limit of instances. The best would be to keep the messages in a queue until the instances free up.

Most of the answers would probably suggest increasing the limit of instances. We already set 1000. This would not be scalable because even if we set the limit to 1500 and a huge spike comes, we would pass the limit and get the 429 messages again.

The only option I can think of is some flow control. So far, we have read about Cloud Tasks, but we are not sure if this can help us. Ideally, we don't want to introduce any new service, but if necessary, we will do.

Thank you for all your tips and time! :)

Here are some options:

1. Use a 1st gen event-driven Cloud Function https://stackoverflow.com/a/75763414/10720618
2. Use Cloud Tasks to rate limit. But then you don't get dead lettering
3. Disable dead lettering and have pubsub continuously attempt delivery. May want to set an end condition to avoid infinite retry loops. Also need to setup some alerting to ensure messages don't expire (if the service cannot keep up with the load).
4. Handle in application code by tracking attempt count for each messageId (IE using Redis), and publishing to a dead letter topic when attempt count has exceeded a threshold. This is easy to abstract, doesn't introduce a new service, and checks all the boxes but definitely adds overhead of state management.