apache-camelspring-camelcamel-ftpapache-camel-aws

How to have disposable routes in Camel?


Usecase: We generate a csv file when user hits an endpoint, which we want to upload to different sources (FTP & S3). We need to encrypt & transform the files before uploading. Here's how the routes look like (oversimplified):

FTP Route:

from("file:temp?include=sample.csv&noop=true")
.routeId("ftpRoute" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("sftp:localhost:22/destinationDir?username=username&password=RAW(password)")
.process(new KillRouteProcessor());

S3 Route:

from("file:temp?include=sample.csv&noop=true")
.routeId("s3Route" + LocalDateTime.now())
.marshal().pgp("file:temp/encryptionKey.asc", "someuserid", null, true, true)
.process(new SomeComplexProcessor())
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)&region=REGION")
.setHeader(FILE_NAME, "metadata.txt")
.process(new MetadataFileGenerator())
.marshal()
.bindy(BindyType.Csv, MetadataFile.class)
.to("aws-s3:bucketName?accessKey=ACCESS_KEY&secretKey=RAW(SECRET_KEY)&region=REGION")
.process(new KillRouteProcessor());

What's working: S3 & SFTP upload routes are working correctly.

Requirements:

  1. It'd be great if the code can be shared. Some different parameters / processors could be present in both routes.
  2. File needs to be deleted after the execution of both routes.
  3. We need to kill the routes after upload, as each request will have a different filename.
  4. Above two routes cannot be merged into one, as there are too many if-else conditions inside the actual routes, which'd further complicate this.
  5. Routes should be optional (eg. Choice of upload to s3/sftp or both should be available)

What I've tried:

  1. Camel Direct: It helps with code reuse, but it doesn't allow multiple consumers (In my case these two routes)
  2. Camel Seda: It allows multiple consumers, but it doesn't seem to allow synchronous routes.
  3. Deleting the file outside of camel context. This is a problem, as we won't know how much time will the routes take to upload files.

Environment:
Camel 3.4.3, Spring Boot 2.2.3, Java8,


Solution

  • I managed to get it working by using a static route id (Removed LocalDateTime.now()). Here's what I learnt while fixing Camel issues.

    1. Always provide route ids to every route. Especially while using 'direct' routes.
    2. Never use dynamic route id. This is very important. I was seeing this issue when I had used LocalDateTime.now() in my route id. I was seeing this error before I changed:

    Multiple consumers for the same endpoint is not allowed: direct://routeName .....

    1. When using loops or calling any direct routes, always use enrich.

    eg. .enrich("direct:subroute", AggregationStrategies.useOriginal()).
    This will share a copy of parent route's headers to the subroute. It'll help you to avoid some weird issues.

    Feel free to comment in case if you'd like to know more.