google-cloud-platformgoogle-bigqueryworkflowdataform

Issue while orchestrating a Dataform Workflow through GCP Workflows


I'm trying to schedule Dataform executions with Workflows and Cloud Scheduler, as documented here.

Specifically, I'm trying to customize the Dataform workflow invocation request in order to execute only those models with the 'hourly' tag, this part is documented here.

The resulting YAML I have is:

main:
    steps:
    - init:
        assign:
        - repository: projects/XXXXX/locations/europe-west4/repositories/XXXXX
    - createCompilationResult:
        call: http.post
        args:
            url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/compilationResults"}
            auth:
                type: OAuth2
            body:
                compilationResult: ${compilationResult.body.name}                
                invocationConfig:
                    includedTags:
                    - hourly
                    transitiveDependenciesIncluded: true
        result: compilationResult
    - createWorkflowInvocation:
        call: http.post
        args:
            url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/workflowInvocations"}
            auth:
                type: OAuth2
            body:
                compilationResult: ${compilationResult.body.name}
        result: workflowInvocation
    - complete:
        return: ${workflowInvocation.body.name}

But I get this error when I try to deploy this workflow:

Could not deploy workflow: failed to build: error in step createCompilationResult: error evaluating attribute for key 'body': error evaluating attribute for key 'compilationResult': symbol 'compilationResult.body.invocationConfig' is undefined, and symbol 'compilationResult.body' is undefined, and symbol 'compilationResult' is neither a variable nor a sub-workflow name (Code: 3)

I don't have many experience in how to solve this. I'm just following the linked documentation and apparently I'm missing something or the documentation is wrong.

Do you know how to solve this?


Solution

  • I think the documentation is wrong. I obtained what I wanted, but not following what suggested the doc here: The doc suggests to:

    replace the createCompilationResult body with the following code snippet...

    But I actually replaced the same code snippet in the createWorkflowInvocation, just because it had more sense... and it worked as I expected. I think there is a Copy/Paste issue in the documentation, copying and pasting from the previous section to this one.

    I'm sharing the correct YAML (at least, what worked for me):

    main:
        steps:
        - init:
            assign:
            - repository: projects/XXXXX/locations/europe-west4/repositories/XXXXX
        - createCompilationResult:
            call: http.post
            args:
                url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/compilationResults"}
                auth:
                    type: OAuth2
                body:
                    gitCommitish: main                
            result: compilationResult
        - createWorkflowInvocation:
            call: http.post
            args:
                url: ${"https://dataform.googleapis.com/v1beta1/" + repository + "/workflowInvocations"}
                auth:
                    type: OAuth2
                body:
                    compilationResult: ${compilationResult.body.name}
                    invocationConfig:
                        includedTags:
                        - hourly
                        transitiveDependenciesIncluded: true
            result: workflowInvocation
        - complete:
            return: ${workflowInvocation.body.name}