I'm using Rails 5 on Heroku for a very lean payment processing app that will be doing large volumes of processing.
The app doesn't have much functionality, which is why it's being hosted there. However, one thing it needs to do 100% reliably though is run payment processing jobs, and run certain ones only a single time. If not 100% (because nothing is perfect), than 99.9999%.
I chose Delayed Job over Sidekiq for this extra database integrity. I also was planning to use the Clockwork gem.
After reading issues though it seems there's a lot of issues with clock processes being ran many times when dynos restart.
I'm wondering what is the best way to solve this issue with Rails/Heroku. How can I very, very reliably run certain jobs only a single time, despite restarts and whatever else?
Also, I forgot to mention that some jobs can be re-run multiple times. It's likely those jobs will be the ones set to run daily or whenever. I'm not concerned with them really.
I am also talking about however jobs schedule from within a Rails action (those will be the critical ones). Is it possible a web dyno restart will prevent Delayed Job from scheduling a job from within my code if it restarts mid-action?
With a state machine. You set the job's state to :pending when the job is created. When the job begins to run change the state to :running. When the job completes change the state to :complete. Only start jobs when they are in the :pending state and have a separate batch processor that checks for :running jobs that have reached a time out. That way when jobs have been killed before completing, you can restart them.