I'm providing both a staging and temp location via the gcloud dataflow flex-template run
CLI. These are both valid optional flags (as per docs) and there isn't any mention that you should or shouldn't provide both.
Why else would this error come up?
ERROR: (gcloud.dataflow.flex-template.run) unrecognized arguments: --temp-location (did you mean '--staging-location'?) gs://gcs-bucket-name
Edit: adding context as requested
The process is executing within a Buildkite CI/CD pipeline, so generally speaking a Buildkite agent/step calls a gcloud Docker container which runs a bash script. I'm also able to run this command and container combo locally just fine -- the error is only present while running in the pipeline
Dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk
COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh
WORKDIR /app
# RUN sudo apt-get install google-cloud-sdk <-- threw error
ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"
gcloud-deploy-flex-template.sh
gcloud dataflow flex-template run ${JOB_NAME} \
--template-file-gcs-location ${TEMPLATE_PATH}.json \
--region us-central1 \
--staging-location ${GCS_PATH}/staging/${JOB_NAME} \
--temp-location ${GCS_PATH}/temp \
--parameters requirements_file=requirements.txt \
--parameters input_subscription=${INPUT_SUBSCRIPTION} \
--parameters output_table=${OUTPUT_TABLE} \
--parameters subject=${SUBJECT} \
--parameters schema_registry_url=${SCHEMA_REGISTRY_URL} \
--subnetwork=${SUBNETWORK} \
--service-account-email=${SERVICE_ACCOUNT_EMAIL}
As I proposed in my comment, you can use the offical google
sdk
image with the latest version :
FROM google/cloud-sdk:412.0.0
COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh
WORKDIR /app
# RUN sudo apt-get install google-cloud-sdk <-- threw error
ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"
or
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1
COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh
WORKDIR /app
# RUN sudo apt-get install google-cloud-sdk <-- threw error
ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"