streamsets

Add text annotation to SDC pipeline?

Is it possible to add a text boxes (/annotations) to an SDC pipeline (v3.8.2)? I want to be able to write a note describing what a few different parts of the pipeline are doing.

Solution

  • You can use Fragments if you use streamsets control hub. And give them a custom names.

    https://streamsets.com/documentation/controlhub/3.13.x/onpremhelp/datacollector/UserGuide/Pipeline_Fragments/PipelineFragments_title.html

    If you don't use control hub, you can try to use Orchestration stages to call another pipelines. https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html#Orchestrators_Title

    So basically split your logic into several pipelines, and create a "orchestration" pipeline that will call them.