google-cloud-platformgoogle-cloud-dataproc

Filtering GCP dataproc batches list by createTime doesn't work

I am trying to get a list of GCP dataproc batches that have been created (succeeded/failed/running) on a specific date. But, I keep getting errors when trying to filter the results of dataproc batches list command using a time field.

I followed Google Cloud's documentation here and tried several different commands:

gcloud dataproc batches list --region=us-central1 --limit=1 --filter="createTime.date('%Y-%m-%d')='2023-11-01'"
gcloud dataproc batches list --region=us-central1 --limit=1 --filter="createTime.date('%Y-%m-%d',Z)='2023-11-01'"
gcloud dataproc batches list --region=us-central1 --limit=1 --filter="stateTime.date('%Y-%m-%d')='2023-11-01'"
gcloud dataproc batches list --region=us-central1 --limit=1 --filter="stateTime.date('%Y-%m-%d',Z)='2023-11-01'"

But, all of the commands throw an error

ERROR: (gcloud.dataproc.batches.list) INVALID_ARGUMENT: Filter string 'createTime.date('%Y-%m-%d)>2023-11-01' is not a valid filter.

I followed Google Cloud's documentation here to identify resource keys. I tried listing all resource-keys for dataproc batches and both createTime and stateTime are resource-keys. But, when I tried

gcloud dataproc batches list --region=us-central1 --limit=1 --filter="createTime='2023-11-01'"
gcloud dataproc batches list --region=us-central1 --limit=1 --filter="stateTime='2023-11-01'"

I get an unsupported field error:

ERROR: (gcloud.dataproc.batches.list) INVALID_ARGUMENT: unsupported field 'createTime'

What am I doing wrong ?

Solution

  • You have an error in the filter syntax (camel case vs snake case in filter field names), here is how you can filter out batches using gcloud:

    gcloud dataproc batches list \
  --limit=5 \
  --filter='create_time<"2023-10-10" AND create_time>"2023-10-01"' \
  --format='table(name,create_time)'