The rate limit set as 600 req/min by default is very low for our application where we are trying to process millions of records through a Spark job. Even with cluster of 16 nodes and 4 cores we are hitting the rate limit. We plan to have 25 such jobs running in parallel. Can you please suggest how the rate limit can be increased to something like 20k per min
According with this official GCP documentation:
You can edit your quotas up to their maximum values by selecting Edit Quotas from the Quotas page of the Google Cloud Dashboard. To request an increase in quota, edit your quota with your requested increase and justification and submit your update. You are notified when your request is received. You might be contacted for more information regarding your request. After your request is reviewed, you are notified whether it has been approved or denied.