I'm trying to schedule Dataform executions with Workflows and Cloud Scheduler, as documented here.
Specifically, I'm trying to customize the Dataform workflow invocation request in order to execute only those models with the 'hourly' tag, this part is documented here.
The resulting YAML I have is:
main:
steps:
- init:
assign:
- repository: projects/XXXXX/locations/europe-west4/repositories/XXXXX
- createCompilationResult:
call: http.post
args:
url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/compilationResults"}
auth:
type: OAuth2
body:
compilationResult: ${compilationResult.body.name}
invocationConfig:
includedTags:
- hourly
transitiveDependenciesIncluded: true
result: compilationResult
- createWorkflowInvocation:
call: http.post
args:
url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/workflowInvocations"}
auth:
type: OAuth2
body:
compilationResult: ${compilationResult.body.name}
result: workflowInvocation
- complete:
return: ${workflowInvocation.body.name}
But I get this error when I try to deploy this workflow:
Could not deploy workflow: failed to build: error in step createCompilationResult: error evaluating attribute for key 'body': error evaluating attribute for key 'compilationResult': symbol 'compilationResult.body.invocationConfig' is undefined, and symbol 'compilationResult.body' is undefined, and symbol 'compilationResult' is neither a variable nor a sub-workflow name (Code: 3)
I don't have many experience in how to solve this. I'm just following the linked documentation and apparently I'm missing something or the documentation is wrong.
Do you know how to solve this?
I think the documentation is wrong. I obtained what I wanted, but not following what suggested the doc here: The doc suggests to:
replace the createCompilationResult body with the following code snippet...
But I actually replaced the same code snippet in the createWorkflowInvocation, just because it had more sense... and it worked as I expected. I think there is a Copy/Paste issue in the documentation, copying and pasting from the previous section to this one.
I'm sharing the correct YAML (at least, what worked for me):
main:
steps:
- init:
assign:
- repository: projects/XXXXX/locations/europe-west4/repositories/XXXXX
- createCompilationResult:
call: http.post
args:
url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/compilationResults"}
auth:
type: OAuth2
body:
gitCommitish: main
result: compilationResult
- createWorkflowInvocation:
call: http.post
args:
url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/workflowInvocations"}
auth:
type: OAuth2
body:
compilationResult: ${compilationResult.body.name}
invocationConfig:
includedTags:
- hourly
transitiveDependenciesIncluded: true
result: workflowInvocation
- complete:
return: ${workflowInvocation.body.name}