shellscriptingstreamsets

Shell to run StreamSets Pipeline


I want to connect to the Control Hub and want to run the particular StreamSets pipeline using Shell Script.


Solution

  • In the StreamSets Control Hub web UI, click on RESTful API and then Job Runner. You will see instructions for authenticating from the command line using curl. Scroll down and you will see the API for starting a job given the job id.

    Putting them together, you'll need to do something like:

    # login to Control Hub security app
    curl -X POST -d '{"userName":"SCHUserID", "password": "SCHUserPassword"}' \
    https://cloud.streamsets.com/security/public-rest/v1/authentication/login \
    --header "Content-Type:application/json" --header "X-Requested-By:SCH" \
    -c cookie.txt
    
    # generate auth token from security app
    sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)
    echo "Generated session token : $sessionToken"
    
    # Start a job using the auth token
    curl -X GET https://cloud.streamsets.com/jobrunner/rest/v1/job/{jobId}/start \
    --header "Content-Type:application/json" --header "X-Requested-By:SCH" \
    --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" \
    -i