djangogitsynchronizationdistributed-computingdjango-q

What's an efficient way of synchronizing git pull requests and restarting systemctl services for multiple Django-Q clusters?

I'm running numerous Django-Q clusters (on Ubuntu focal) to perform distributed computing on large data sets. I have production clusters and development clusters that each have their own git branch. I need a way to synchronize the updating and restarting of these cluster systemctl services when changes to their respective git repositories are made.

Currently I ssh into each machine and run the following commands.

$ git pull
$ sudo systemctl restart qcluster

I appreciate any ideas you may have.

Solution

  • The problem with this resides in its on-demand aspect.
    You need to detect somehow there are new commits, and then initiate your SSH+pull+restart.

    Depending on your remote hosting service (GitHub/GitLab,...), you would set a webhook listener on each reachable, to automate that step.
    Once the listener receive notice of a new commit, it can in turn connect to the clusters and operate your commands.
    An Ansible playbook would be a good fit, since Ansible is agentless, and relies mainly on SSH to execute scripts on remote machines.