javaspringspring-batchspring-cloud-streamspring-cloud-dataflow

Spring Cloud data flow docker connection


I have created a Spring Batch app and packaged it as a JAR file. However, when I try to upload it to SCDF (running inside a Docker container), SCDF can't access the JAR file, and I'm not sure how to make the connection between SCDF (Docker) and my JAR file.


Solution

  • Apps running in SCDF are identified by name, type, version, and location URI. The location URI is the "connection between SCDF (Docker) and my JAR file" and can have a scheme of maven://, docker://, http://, or file://.

    Maven

    In a local install of SCDF running in Docker you can register your app using a maven:// location URI pointing to the jar file in your local Maven repo. You can get the jar file into the repo by running mvn install when using Maven or gradle publishToMavenLocal when using Gradle.

    Since SCDF is running in Docker, the containers need access to the local filesystem in order to access the local Maven repo. If you are using the Docker Compose files from SCDF you just need to export the following env vars before starting SCDF:

    export HOST_MOUNT_PATH=~/.m2
    export DOCKER_MOUNT_PATH=/root/.m2/
    

    However, if you are using your own custom Docker Compose files then you will need to add a volume mount to your service in your compose file like this:

    services:
      dataflow-server:
        volumes:
          - ~/.m2:/root/.m2/
    

    File

    To use file:// you will need to configure the mount paths using the same process described previously in the Maven section. More details here.

    Https

    To use https:// you will need to host your jar file on a web server that is accessible by the containers.

    Docker

    To use docker:// you will have to jump through some hoops via Docker-out-of-Docker (DooD) as described here.