[SOLVED] How can I debug an InactiveRpcError with status INVALID_ARGUMENT when submitting a serverless batch job?

How can I debug an InactiveRpcError with status INVALID_ARGUMENT when submitting a serverless batch job?

When submitting a dataproc serverless batch request, we have been getting errors like:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "Request contains an invalid argument."
    debug_error_string = "UNKNOWN:Error received from peer ipv4:**** {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-09-23T20:08:11.385708286+00:00"}"
>

These errors appears to be transient as retries with the same arguments succeed. This problem began for us late last week and we have not changed anything in the request or package versions. We are currently using:

apache-airflow-providers-google (10.21.1) to submit jobs in an Airflow composer environment
google-cloud-dataproc (5.6.0) to submit jobs via the Python SDK

And have been experiencing failures with both.

We have tried updating versions and faced the same issues. We have added retries to our requests and this works in some cases but it is still unclear why a request of the same arguments would fail once with an invalid argument error but succeed in a subsequent retry.

Solution

After reaching out to our Google reps, they informed us they identified a possibly related bug and deployed a fix. We have not noticed the issue since.