[SOLVED] Python as a service in docker or cron?

Python as a service in docker or cron?

What is the python script doing?

I have implemented leader election for Artemis ActiveMQ Primary-Backup HA. This checks the availability of ActiveMQ pods and updates the labels in the pod(The label of service is fixed to 'primary'). This helps the service to route the traffic to the current 'primary' pod.

Pseudocode:

while true:
  if currentPod's TCP socket 61616 is running:
    change the label of the pod to 'primary' if not already 'primary'
  else:
    change the label of the pod to 'backup' if not already 'backup'
  wait(3 sec)

Pod's base image: eclipse-temurin:17

For me, this works fine, and once the primary pod crashes or restarts, all incoming traffic is sweetly shifted to the second pod.

How do I run this script?

In entrypoint.sh -

python3 $PATH/leader-election.py &

Dockerfile & entrypoint.sh

These files have n number of things so I wont be adding it all, but here is part of it important for this question

# Create broker instance
FROM eclipse-temurin:17 as builder
ARG ACTIVEMQ_ARTEMIS_VERSION
ENV ACTIVEMQ_ARTEMIS_VERSION=$ACTIVEMQ_ARTEMIS_VERSION
RUN apt-get -qq -o=Dpkg::Use-Pty=0 update && apt-get -y install  && apt-get -y install wget 
RUN apt-get install -y python3
RUN apt-get install -y python3-pip

RUN wget https://archive.apache.org/dist/activemq/activemq-artemis/$ACTIVEMQ_ARTEMIS_VERSION/apache-artemis-$ACTIVEMQ_ARTEMIS_VERSION-bin.tar.gz

RUN tar xvfz apache-artemis-$ACTIVEMQ_ARTEMIS_VERSION-bin.tar.gz
RUN ln -s /opt/apache-artemis-${ACTIVEMQ_ARTEMIS_VERSION} /opt/apache-artemis
RUN rm -f apache-artemis-${ACTIVEMQ_ARTEMIS_VERSION}-bin.tar.gz KEYS apache-artemis-${ACTIVEMQ_ARTEMIS_VERSION}-bin.tar.gz.asc

WORKDIR /var/lib
RUN  "/opt/apache-artemis-${ACTIVEMQ_ARTEMIS_VERSION}/bin/artemis" create artemis \
        --home /opt/apache-artemis \
        --http-host 0.0.0.0 \
        --user artemis \
        --password artemis \
        --no-hornetq-acceptor --no-mqtt-acceptor --no-stomp-acceptor --no-amqp-acceptor \
        --role amq \
        --require-login ;
    
# some more configs
COPY assets/docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["artemis-server"]

if [ "$LEADER_ELECTION" = "true" ]; then
  python3 $OVERRIDE_PATH/leader-election.py &
fi
# some more things
if [ "$1" = 'artemis-server' ]; then
  exec dumb-init -- sh ./artemis run
fi

Statefulset and Service

apiVersion: v1
kind: Service
metadata:
  name: activemq-svc
  namespace: activemq
spec:
  type: ClusterIP
  ports:
  - port: 61616
    name: netty-connector
    protocol: TCP
    targetPort: 61616
  selector:
    app: activemq
    node: primary

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: activemq-statefulset
  namespace: activemq
spec:
  replicas: 2
  serviceName: activemq-svc
  selector:
    matchLabels:
      app: activemq
  template:
    metadata:
      labels:
        app: activemq
        node: not-ready

The python takes care of changing the node: not-ready to node: primary or node: backup according to the availability. So at a given point of time one of the node is always primary which then matches to service, hence allowing traffic to flow to only that pod.

Problem

This script randomly stops and the leader election becomes a disaster for me. I have been looking for ways on how to run python script as a service, or is cron an option for running the code every 3 seconds? Also, I found How to make a Python script run like a service or daemon in Linux where the requirement is kinda similar. This brings another question, can this be done without an infinite loop?

Solution

I used supervisord to run the python script.