I am running a shell script as Ubuntu's systemd service to start at boot-time. The script internally executes a Python script (python_simulator.py) that connects to InfluxDB (through Python's influxdb package).
The Python script fails to start at boot-time and checking the logs suggests that's because of 'ConnectionError' while connecting to InfluxDB. I interpreted it as it's possible that the influxdb service as not started by the time the Python service activates at boot time. So I have tried to add the order dependency in the service by adding "After" and "Wants" as "influxdb.service" which activates the Python service a few seconds after influxdb service. But, I still get the same connection error.
The systemd service (myservice.service) looks like:
[Unit]
Description= Python startup service.
After=influxdb.service
Wants=influxdb.service
[Service]
Type=forking
ExecStart=/bin/bash /home/test_user/Deploy/start.sh
ExecStop=/bin/bash /home/test_user/Deploy/stop.sh
[Install]
WantedBy=multi-user.target
The log file of the Python script (python_simulator.py):
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection: [$
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "python_simulator.py", line 467, in <module>
main(host=args.host, port=args.port)
File "python_simulator.py", line 312, in main
for db_dict in client.get_list_database():
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 570, in get_list_database
return list(self.query("SHOW DATABASES").get_points())
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 416, in query
expected_response_code=expected_response_code
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 267, in request
timeout=self._timeout
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection$
Lastly, blocking tree of daemons shows that myservice.service executes after influxdb.service:
myservice.service +6.872s └─influxdb.service @11.344s └─network-online.target @11.337s └─NetworkManager-wait-online.service @4.706s +6.630s └─NetworkManager.service @3.940s +674ms └─dbus.service @3.914s └─basic.target @3.728s └─sockets.target @3.728s └─snapd.socket @3.722s +5ms └─sysinit.target @3.712s └─apparmor.service @3.276s +435ms └─local-fs.target @3.266s └─run-user-1000.mount @49.841s └─swap.target @3.160s └─dev-disk-by\x2duuid-16e1b46a\x2d79fc\x2d4965\x2d9932\x2d8f589e9e7057.swap @3.132s +23ms └─dev-disk-by\x2duuid-16e1b46a\x2d79fc\x2d4965\x2d9932\x2d8f589e9e7057.device @3.130s
I am not sure why I am still not able to execute the script (python_simulator.py) with influxdb in place. Are there any other dependencies? Are any changes needed in myservice.service? Any help will be appreciated.
Edit 1:
Can the reason be ConnectionRefusedError instead of ConnectionError and that may be because by the time it connects to influx at Port 8086, nothing is listening on it? If so, how can I put that in dependency order?
You may still run into a problem even if you create the service dependency (because it's possible that influxdb takes a moment before it is ready to accept connections, during which time the Python code might start). The solution is to either:
Without seeing your Python code it's hard to suggest what the first solution might look like, but configuring your unit to restart on failure is as simple as adding the following to your [Service] section:
Restart=on-failure
You can delay startup of your Python code until influxdb is ready by adding a small shell script to your systemd unit:
ExecStartPre=/bin/sh -c 'while ! curl -sf http://localhost:8086/ping; do sleep 1; done'
This will loop indefinitely (thus preventing the service from starting) until influxdb is responding successfully to the /ping endpoint.