I am currently facing an issue with using the dataproc serverless option submission in dbt. My dataproc job takes around 30 minutes to complete, but my dbt runs keep failing with the error message "Operation did not complete within the designated timeout of 900 seconds." The dbt run stops but the dataproc serverless jobs continues running and finishes with no issues.
I have gone through the code: https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/python_submissions.py#L51 and noticed that there is a possibility to change the timeout value. I have tried defining the timeout value in my profile, model config, and in the dbt.config object in the python model itself, but with no success. I have also changed my job_execution_timeout_seconds value to 6000, but that did not work either.
Does anyone have any ideas on how to override the default timeout value in this scenario? Any help would be much appreciated.
Thank you
After upgrading to dbt 1.6.0, the timeout parameter is working with no issues.