node.jsamazon-web-servicescronamazon-sqsamazon-swf

Scalable cron architecture on top of AWS

We have web application used by customers and they have option to create reports. Report contains email and scheduled time(Ex: every day, 9 a.m.).

When the scheduled time matches the current time web app does some work and sends result to the email.

I have one requirement where I need to implement scalable cron architecture on top of swf.

My needed architecture looks like below:

  1. User creates report (DONE)
  2. Webapp saves report to the db and sends report data and timer to the cron microservice through SQS(simple query service). (DONE)
  3. Cron microservice reads incoming SQS messages and sends back SQS message when timer elapsed. (NEED THIS)
  4. Webapp reads SQS message and triggers Data Analyzer and Emailer function to send analyzed data. (DONE)

From I read about SWF service is we can create cron jobs and SWF autoscales. How can I create scalable cron microservice using SWF?

Open to any suggestions...

P.S. Web app is written in nodejs, it will be very good to write microservice in nodejs.

Update 1: After spending some time researching possible solutions, I found https://github.com/capside/CloudCron project. But it relies on cloudwatch event. For many scheduled tasks it may charge a lot IMO.

Update 2: Banjo Obayomi suggested to use Lambda function with SQS and CWE.

Solution 1:

  1. Webapp sends SQS message.
  2. SQS message triggers Lambda function1
  3. Lambda function creates CWE rule
  4. CWE rule triggers Lambda function2
  5. Lambda function2 sends SQS message back to webapp.

Limitations: We can only create 100 CWE rules per region. It means webapp can not generate more than 100 reports with scheduled date.

Links: cron-in-aws-with-lambda-function

Solution

  • I did not find the exact solution to my problem, but found another way to achieve my goal with CloudWatchEvent and SQS services.

    Instead of receiving individual cron SQS messages from cron microservice with params, and doing analyzing, notification work. We moved logic to backend app like below:

    1. Created SQS queue ("cron-microservice-dev") which receives messages from CloudWatchEventRule
    2. Created CloudWatchEventRule that sends SQS queue every 1 hour( we can change timer with cron like syntax). So every 1 hour CWE rule sends message to SQS queue.
    3. Now backend listens to "cron-microservice-dev" queue and checks from DB if there exists scheduled reports that should be triggered. If there exists reports it will analyze each report and notifies the receiver.