pythondockerinfluxdbinfluxdb-python

InfluxDB 2.0 Client Stops Writing After Few Hours (Notifications Still Received, Container Restart Fixes Temporarily)


I’m using InfluxDB 2.0 in Docker along with a custom notification receiver (influx-adapter-api container) that writes to InfluxDB using the Python influxdb-client in synchronous mode.

So it seems likely the issue is with the InfluxDB client or the runtime inside the influx-adapter-api container.

What I’ve Tried

  1. -Switching between SYNCHRONOUS and ASYNCHRONOUS modes
  2. -Reconnecting every hour
  3. -Handling session timeouts manually
  4. -Verifying the container continues receiving real-time data
  5. -Removing excess logging

Question:

influxdb:

  image: influxdb:2.0
  restart: unless-stopped
  expose:
    - "8086"
  ports:
    - "8086:8086"  # For testing only
  environment:
    - DOCKER_INFLUXDB_INIT_MODE=setup
    - DOCKER_INFLUXDB_INIT_USERNAME=influxdb
    - DOCKER_INFLUXDB_INIT_PASSWORD=influxdb
    - DOCKER_INFLUXDB_INIT_ORG=Polito
    - DOCKER_INFLUXDB_INIT_BUCKET=test
    - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=password
  volumes:
    - influxdb_data:/var/lib/influxdb2

writer container:

influx-adapter-api:
      image: influx-adapter-api:latest
      restart: unless-stopped
      expose:
        - "8008"
      volumes:
        - setting:/influx-adapter-api

the code:

from influxdb_client import InfluxDBClient
from influxdb_client.client.write_api import SYNCHRONOUS
from datetime import datetime
import time, threading, logging

class InfluxWriter:
    def __init__(self, url, org, bucket, username, password):
        self.client = InfluxDBClient(url=url, username=username, password=password, org=org)
        self.write_api = self.client.write_api(write_options=SYNCHRONOUS)
        threading.Thread(target=self._periodic_reconnect, daemon=True).start()

    def _periodic_reconnect(self):
        while True:
            time.sleep(3600)
            self._reconnect()

    def _reconnect(self):
        self.write_api.close()
        self.client.close()
        self.client = InfluxDBClient(url=self.url, username=self.username, password=self.password, org=self.org)
        self.write_api = self.client.write_api(write_options=SYNCHRONOUS)

    def write_notification_to_influxdb(self, measurement, fields, timestamp):
        data = {"measurement": measurement, "fields": fields, "time": timestamp}
        try:
            self.write_api.write(bucket=self.bucket, record=data)
            self.write_api.flush()
        except Exception as e:
            if "session not found" in str(e):
                self._reconnect()
                self.write_api.write(bucket=self.bucket, record=data)
                self.write_api.flush()
            else:
                logging.error(f"Write error: {e}")

Solution

  • I tried using a token instead of user/pass, and it's been working longer, but I need to keep an eye on it. However, this does not explain why my other services are working; only this one is disconnecting and is not showing any errors.

    First, I created a token with

    influx auth create \
      --org <ORG_NAME> \
      --read-buckets \
      --write-buckets \
      --description "Token for My App"
    
    and then
    
    InfluxDBClient(url="http://localhost:8086", token="my-token", org="my-org")