Dockerfile executes python script not as expected
For a university assignment I need to migrate a postgres dump to couchbase. The whole process has to run within docker services. At the end the whole migration should be executed by running docker compose.
So far the postgres and couchbase initialisation work correctly.
After the initialisation of the databases I would like to do the actual migration by a python script. The service that should run the script always exits with code 0 but nothing happens. That's why I'm pretty sure that it's not executed correctly.
Working directory:
Dockerfile:
FROM python:3.9
# Set the working directory in the container to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . /app
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# Run script when the container launches
CMD ["python", "./migration_script.py"]
docker-compose:
version: "3.9"
networks:
backend:
external: true
services:
postgres_db:
container_name: postgresdb
image: postgres:13
environment:
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "1234"
POSTGRES_DB: "dvdrental"
ports:
- "5432:5432"
networks:
- "backend"
postgres_seed:
container_name: postgresseed
image: postgres:13
environment:
POSTGRES_HOST_AUTH_METHOD: "trust"
PGPASSWORD: "1234"
volumes:
- ./postgres/dvdrental.tar:/tmp/dvdrental.tar:ro
entrypoint: [ "bash", "-c", "sleep 10 && pg_restore -h postgresdb -U postgres -d dvdrental /tmp/dvdrental.tar" ]
restart: "no"
networks:
- "backend"
depends_on:
- "postgres_db"
adminer:
container_name: adminer
image: adminer
networks:
- "backend"
ports:
- "${UI_PORT?}:8080"
couchbase_server:
container_name: couchbase
image: couchbase:7.1.3
ports:
- "8091-8094:8091-8094"
- "11210:11210"
networks:
- backend
couchbase_init:
container_name: couchbase_init
image: couchbase:7.1.3
volumes:
- ./init-couchbase.sh:/opt/init-couchbase.sh
depends_on:
- couchbase_server
networks:
- backend
command: /bin/sh -c "chmod +x /opt/init-couchbase.sh && /opt/init-couchbase.sh"
data_migration_service:
container_name: data_migration
image: alpine
build: .
depends_on:
- couchbase_init
networks:
- backend
To check whether the script is executed or not I changed the actual migration script to the following which just checks whether couchbase is running:
migration_script.py:
import time
import requests
COUCHBASE_HOST = "couchbase" # or the appropriate host
COUCHBASE_PORT = "8091"
COUCHBASE_URL = f"http://{COUCHBASE_HOST}:{COUCHBASE_PORT}/pools"
print("Waiting for Couchbase Server to start...")
while True:
try:
response = requests.get(COUCHBASE_URL)
if response.status_code == 200:
print("Couchbase Server is up and running.")
break
except requests.ConnectionError:
print("Couchbase Server is not available yet, waiting...")
time.sleep(1)
requirements.txt:
psycopg2-binary==2.9.3
couchbase==4.1.8
requests==2.31.0
I already tried to read the logs of the data_migration container. They seem to be empty. That's why I'm pretty sure that the script has never been really executed.
Here are the logs created by running docker compose up:
[+] Building 0.0s (0/0) docker:desktop-linux
[+] Running 6/6
✔ Container couchbase Created 0.0s
✔ Container postgresdb Created 0.0s
✔ Container adminer Created 0.0s
✔ Container couchbase_init Created 0.0s
✔ Container postgresseed Created 0.0s
✔ Container data_migration Created 0.0s
Attaching to adminer, couchbase, couchbase_init, data_migration, postgresdb, postgresseed
couchbase | Starting Couchbase Server -- Web UI available at http://<ip>:8091
couchbase | and logs available in /opt/couchbase/var/lib/couchbase/logs
adminer | [Thu Nov 30 09:09:08 2023] PHP 7.4.33 Development Server (http://[::]:8080) started
postgresdb | The files belonging to this database system will be owned by user "postgres".
postgresdb | This user must also own the server process.
postgresdb |
postgresdb | The database cluster will be initialized with locale "en_US.utf8".
postgresdb | The default database encoding has accordingly been set to "UTF8".
postgresdb | The default text search configuration will be set to "english".
postgresdb |
postgresdb | Data page checksums are disabled.
postgresdb |
postgresdb | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgresdb | creating subdirectories ... ok
postgresdb | selecting dynamic shared memory implementation ... posix
postgresdb | selecting default max_connections ... 100
postgresdb | selecting default shared_buffers ... 128MB
postgresdb | selecting default time zone ... Etc/UTC
postgresdb | creating configuration files ... ok
couchbase_init | Waiting for Couchbase Server to start...
couchbase_init | % Total % Received % Xferd Average Speed Time Time Time Current
couchbase_init | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
couchbase_init | curl: (7) Failed to connect to couchbase port 8091 after 1 ms: Connection refused
postgresdb | running bootstrap script ... ok
data_migration exited with code 0
postgresdb | performing post-bootstrap initialization ... ok
postgresdb | syncing data to disk ... ok
postgresdb |
postgresdb | initdb: warning: enabling "trust" authentication for local connections
postgresdb | You can change this by editing pg_hba.conf or using the option -A, or
postgresdb | --auth-local and --auth-host, the next time you run initdb.
postgresdb |
postgresdb | Success. You can now start the database server using:
postgresdb |
postgresdb | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgresdb |
postgresdb | waiting for server to start....2023-11-30 09:09:09.088 UTC [48] LOG: starting PostgreSQL 13.12 (Debian 13.12-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgresdb | 2023-11-30 09:09:09.088 UTC [48] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgresdb | 2023-11-30 09:09:09.091 UTC [49] LOG: database system was shut down at 2023-11-30 09:09:08 UTC
postgresdb | 2023-11-30 09:09:09.093 UTC [48] LOG: database system is ready to accept connections
postgresdb | done
postgresdb | server started
postgresdb | CREATE DATABASE
postgresdb |
postgresdb |
postgresdb | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgresdb |
postgresdb | waiting for server to shut down...2023-11-30 09:09:09.273 UTC [48] LOG: received fast shutdown request
postgresdb | .2023-11-30 09:09:09.273 UTC [48] LOG: aborting any active transactions
postgresdb | 2023-11-30 09:09:09.275 UTC [48] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
postgresdb | 2023-11-30 09:09:09.275 UTC [50] LOG: shutting down
postgresdb | 2023-11-30 09:09:09.281 UTC [48] LOG: database system is shut down
postgresdb | done
postgresdb | server stopped
postgresdb |
postgresdb | PostgreSQL init process complete; ready for start up.
postgresdb |
postgresdb | 2023-11-30 09:09:09.393 UTC [1] LOG: starting PostgreSQL 13.12 (Debian 13.12-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgresdb | 2023-11-30 09:09:09.393 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgresdb | 2023-11-30 09:09:09.393 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgresdb | 2023-11-30 09:09:09.394 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgresdb | 2023-11-30 09:09:09.396 UTC [63] LOG: database system was shut down at 2023-11-30 09:09:09 UTC
postgresdb | 2023-11-30 09:09:09.399 UTC [1] LOG: database system is ready to accept connections
couchbase_init | % Total % Received % Xferd Average Speed Time Time Time Current
couchbase_init | Dload Upload Total Spent Left Speed
100 551 100 551 0 0 103k 0 --:--:-- --:--:-- --:--:-- 107k
couchbase_init | {"isAdminCreds":true,"isROAdminCreds":false,"isEnterprise":true,"allowedServices":["kv","n1ql","index","fts","cbas","eventing","backup"],"isDeveloperPreview":false,"packageVariant":"linux/docker","pools":[],"settings":[],"uuid":[],"implementationVersion":"7.1.3-3479-enterprise","componentsVersion":{"asn1":"5.0.18","os_mon":"2.7.1","lhttpc":"1.3.0","chronicle":"0.0.1","ns_server":"7.1.3-3479-enterprise","kernel":"8.3.2.1","public_key":"1.12.0.1","sasl":"4.1.2","crypto":"5.0.6.3","inets":"7.5.3.1","ssl":"10.7.3.3","stdlib":"3.17.2","ale":"0.0.0"}}SUCCESS: Cluster initialized
couchbase_init | SUCCESS: Bucket created
couchbase_init exited with code 0
postgresseed exited with code 0
I know that the script is executed before the couchbase initialisation is finished but this should not prevent the script from executing and creating some output.
Solution
Omit the image from the data_migration_service.
From the documentation:
Only a subset of Compose file services can be defined under a build subsection, others may make use of the image attribute. When a build subsection is present for a service, Compose ignores the image attribute for the corresponding service, as Compose can build an image from source.
Consistency with image When a service definition includes both the image attribute and a build section, Compose can't guarantee a pulled image is strictly equivalent to building the same image from source. Without any explicit user directives, Compose with build support first tries to pull the image, then builds from source if the image is not found on registry. Compose may offer options to customize this behaviour by user request.
This being in conflict, I think what is happening is you are actually executing an alpine container, which just opens a shell. I am actually not sure why it stops then, but that would be my theory.
- count the elements in a list
- How to solve: ValueError: Invalid format specifier?
- How to generate an MD5 checksum of a file?
- What is the point for asyncio synchronization primitives not to be thread safe?
- How to increase a counter in SQLAlchemy
- How update/remove conda-forge channel from Anaconda?
- Editing specific labels in stacked bar chart with matplotlib
- Strange results with huggingface transformer[marianmt] translation of larger text
- Appending column values from one dataframe to another as a list
- Intersection 3D meshes python
- extract variables from list and run a loop
- Overlay polygon on top of image in Python
- bit-wise operation unary ~ (invert)
- Variable table name in sqlite
- SQL SELECT from table name supplied by a variable
- How do I detect collision in pygame?
- Can you override the default formatter for f-strings?
- Streaming function in Microsoft Autogen
- Importing dynamically all modules from a folder
- Is it possible to type annotate Python function parameter used as TypedDict key to make mypy happy?
- Using VirtualEnv with multiple Python versions on windows
- How to pickle or store Jupyter (IPython) notebook session for later
- Why does a Pygame display with the pygame.FULLSCREEN flag affect other windows?
- color discrete sequence in a plotly graph objects
- How to specify dependencies for the entire router?
- What is the easiest way to share a python environment?
- docker streamlink opens a stream but immediately closes it
- How to attach a Scrollbar to a Text widget?
- Getting today's date in YYYY-MM-DD in Python?
- Get the current git hash in a Python script