EDIT1 31/12/2020
well it doesn't works because django service remain unhealthy need some help
EDIT 31/12/2020
error UniqueViolation (celery-beat) and DuplicateTable (celery) errors with django_migrations using django/postgresql/celery/celery-beat with DOCKER
I am learning about docker-compose healthycheck as depends_on is not sufficient as container only wait for container it depend to be started
If celery and celery-beat "wait" for django web app be running on http://0.0.0.0:8000/ it should prevent error below?
...
django:
restart: always
container_name: django
...
healthcheck:
test: ["CMD", "curl", "-f", "http://0.0.0.0:8000"]
interval: 30s
timeout: 10s
retries: 5
celery:
container_name: celery
build:
context: ./app
dockerfile: Dockerfile.dev
command: celery -A core worker -l info
volumes:
- ./app:/usr/src/app
env_file:
- ./.env.dev
depends_on:
web:
condition: service_healthy
redis:
condition: service_started
EDIT
Error are not the same and deal with auth_permission table now
I develop a Django app and try to implement asynchronous task for dbbackup
stack: django/postgresql/celery/celery-beat and DOCKER (only postgresql is not in a container)
I am new with celery and celery-beat
When run my project, I have 2 errors with celery and celery-beat I do not understand
traces:
D:\Users\jl3\DOCKER\Apps\cafe_tropical>docker-compose -f docker-compose.dev.yml up
Creating network "cafe_tropical_default" with the default driver
Creating redis ... done Creating web ... done Creating celery ... done Creating celery-beat ... done Attaching to redis, celery, web, celery-beat
celery | Waiting for postgres...
celery | PostgreSQL started
celery-beat | Waiting for postgres...
celery-beat | PostgreSQL started
redis | 1:C 30 Dec 2020 15:21:18.345 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis | 1:C 30 Dec 2020 15:21:18.345 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis | 1:C 30 Dec 2020 15:21:18.345 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis | 1:M 30 Dec 2020 15:21:18.346 * Running mode=standalone, port=6379.
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis | 1:M 30 Dec 2020 15:21:18.346 # Server initialized
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis | 1:M 30 Dec 2020 15:21:18.348 * Ready to accept connections
web | Waiting for postgres...
web | PostgreSQL started
celery | No changes detected
celery-beat | No changes detected
web | No changes detected
celery | Operations to perform:
celery | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery | Running migrations:
celery | Applying contenttypes.0001_initial... OK
web | Operations to perform:
web | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
web | Running migrations:
celery-beat | Operations to perform:
celery-beat | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery-beat | Running migrations:
celery | Applying auth.0001_initial... OK
celery-beat | Applying auth.0001_initial...Traceback (most recent call last):
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat | return self.cursor.execute(sql)
celery-beat | psycopg2.errors.UniqueViolation: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat |
celery-beat |
celery-beat | The above exception was the direct cause of the following exception:
celery-beat |
celery-beat | Traceback (most recent call last):
celery-beat | File "manage.py", line 22, in <module>
web | Applying auth.0001_initial...Traceback (most recent call last):
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web | return self.cursor.execute(sql)
web | psycopg2.errors.UniqueViolation: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web |
web |
web | The above exception was the direct cause of the following exception:
web |
web | Traceback (most recent call last):
web | File "manage.py", line 22, in <module>
web | main()
web | File "manage.py", line 18, in main
celery-beat | main()
celery-beat | File "manage.py", line 18, in main
web | execute_from_command_line(sys.argv)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
web | utility.execute()
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
web | self.fetch_command(subcommand).run_from_argv(self.argv)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
web | self.execute(*args, **cmd_options)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
web | output = self.handle(*args, **options)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
web | res = handle_func(*args, **kwargs)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
web | post_migrate_state = executor.migrate(
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
web | state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
web | state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
web | state = migration.apply(state, schema_editor)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
web | operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
web | schema_editor.create_model(model)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
web | self.execute(sql, params or None)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
web | cursor.execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
web | return super().execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
web | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
web | return executor(sql, params, many, context)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
web | return self.cursor.execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
web | raise dj_exc_value.with_traceback(traceback) from exc_value
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web | return self.cursor.execute(sql)
web | django.db.utils.IntegrityError: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web |
celery-beat | execute_from_command_line(sys.argv)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
celery-beat | utility.execute()
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
celery-beat | self.fetch_command(subcommand).run_from_argv(self.argv)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
celery-beat | self.execute(*args, **cmd_options)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
celery-beat | output = self.handle(*args, **options)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
celery-beat | res = handle_func(*args, **kwargs)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
celery-beat | post_migrate_state = executor.migrate(
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
celery-beat | state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
celery-beat | state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
celery-beat | state = migration.apply(state, schema_editor)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
celery-beat | operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
celery-beat | schema_editor.create_model(model)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
celery-beat | self.execute(sql, params or None)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
celery-beat | cursor.execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
celery-beat | return super().execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
celery-beat | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
celery-beat | return executor(sql, params, many, context)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-beat | return self.cursor.execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
celery-beat | raise dj_exc_value.with_traceback(traceback) from exc_value
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat | return self.cursor.execute(sql)
celery-beat | django.db.utils.IntegrityError: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat |
celery | Applying admin.0001_initial... OK
celery | Applying admin.0002_logentry_remove_auto_add... OK
celery | Applying admin.0003_logentry_add_action_flag_choices... OK
celery | Applying api.0001_initial... OK
celery | Applying contenttypes.0002_remove_content_type_name... OK
celery | Applying auth.0002_alter_permission_name_max_length... OK
celery | Applying auth.0003_alter_user_email_max_length... OK
celery | Applying auth.0004_alter_user_username_opts... OK
celery | Applying auth.0005_alter_user_last_login_null... OK
celery | Applying auth.0006_require_contenttypes_0002... OK
celery | Applying auth.0007_alter_validators_add_error_messages... OK
celery | Applying auth.0008_alter_user_username_max_length... OK
celery | Applying auth.0009_alter_user_last_name_max_length... OK
celery | Applying auth.0010_alter_group_name_max_length... OK
celery | Applying auth.0011_update_proxy_permissions... OK
celery | Applying authtoken.0001_initial... OK
celery | Applying authtoken.0002_auto_20160226_1747... OK
celery | Applying cafe.0001_initial... OK
celery | Applying django_celery_beat.0001_initial... OK
celery | Applying django_celery_beat.0002_auto_20161118_0346... OK
celery | Applying django_celery_beat.0003_auto_20161209_0049... OK
celery | Applying django_celery_beat.0004_auto_20170221_0000... OK
celery | Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
celery | Applying django_celery_beat.0006_auto_20180322_0932... OK
celery | Applying django_celery_beat.0007_auto_20180521_0826... OK
celery | Applying django_celery_beat.0008_auto_20180914_1922... OK
celery | Applying django_celery_beat.0006_auto_20180210_1226... OK
celery | Applying django_celery_beat.0006_periodictask_priority... OK
celery | Applying django_celery_beat.0009_periodictask_headers... OK
celery | Applying django_celery_beat.0010_auto_20190429_0326... OK
celery | Applying django_celery_beat.0011_auto_20190508_0153... OK
celery | Applying django_celery_beat.0012_periodictask_expire_seconds... OK
celery | Applying django_celery_beat.0013_auto_20200609_0727... OK
celery | Applying django_celery_beat.0014_remove_clockedschedule_enabled... OK
celery | Applying parameters.0001_initial... OK
celery | Applying sessions.0001_initial... OK
celery | Applying sites.0001_initial... OK
celery | Applying sites.0002_alter_domain_unique... OK
celery-beat | celery beat v5.0.5 (singularity) is starting.
celery-beat | __ - ... __ - _
celery-beat | LocalTime -> 2020-12-30 15:21:35
celery-beat | Configuration ->
celery-beat | . broker -> redis://redis:6379//
celery-beat | . loader -> celery.loaders.app.AppLoader
celery-beat | . scheduler -> celery.beat.PersistentScheduler
celery-beat | . db -> celerybeat-schedule
celery-beat | . logfile -> [stderr]@%INFO
celery-beat | . maxinterval -> 5.00 minutes (300s)
celery-beat | [2020-12-30 15:21:35,216: INFO/MainProcess] beat: Starting...
celery-beat | [2020-12-30 15:21:35,678: INFO/MainProcess] Scheduler: Sending due task hello (cafe.tasks.hello)
celery-beat | [2020-12-30 15:21:35,724: INFO/MainProcess] Scheduler: Sending due task backup (cafe.tasks.backup)
web | Watching for file changes with StatReloader
web | Performing system checks...
web |
web | System check identified no issues (0 silenced).
web | December 30, 2020 - 15:21:37
web | Django version 2.2.5, using settings 'core.settings.dev'
web | Starting development server at http://0.0.0.0:8000/
web | Quit the server with CONTROL-C.
celery | /usr/local/lib/python3.8/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
celery | absolutely not recommended!
celery |
celery | Please specify a different user using the --uid option.
celery |
celery | User information: uid=0 euid=0 gid=0 egid=0
celery |
celery | warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
celery |
celery | -------------- celery@e53033835ccb v5.0.5 (singularity)
celery | --- ***** -----
celery | -- ******* ---- Linux-4.19.128-microsoft-standard-x86_64-with 2020-12-30 15:21:37
celery | - *** --- * ---
celery | - ** ---------- [config]
celery | - ** ---------- .> app: core:0x7fe23b6ba220
celery | - ** ---------- .> transport: redis://redis:6379//
celery | - ** ---------- .> results: redis://redis:6379/
celery | - *** --- * --- .> concurrency: 4 (prefork)
celery | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery | --- ***** -----
celery | -------------- [queues]
celery | .> celery exchange=celery(direct) key=celery
celery |
celery |
celery | [tasks]
celery | . cafe.tasks.backup
celery | . cafe.tasks.hello
celery | . core.celery.debug_task
celery |
celery | [2020-12-30 15:21:37,940: INFO/MainProcess] Connected to redis://redis:6379//
celery | [2020-12-30 15:21:37,951: INFO/MainProcess] mingle: searching for neighbors
celery | [2020-12-30 15:21:38,979: INFO/MainProcess] mingle: all alone
celery | [2020-12-30 15:21:39,009: WARNING/MainProcess] /usr/local/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
but all migrations seems to be running correctly for django web app container
I try to drop postgresql database before building and running containers so I do not understand how django_migrations table could already exist
probably dependencies between containers
I set all containers depending of redis
version: '3.7'
services:
web:
restart: always
container_name: web
build:
context: ./app
dockerfile: Dockerfile.dev
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- redis
redis:
image: "redis:alpine"
celery:
build:
context: ./app
dockerfile: Dockerfile.dev
command: celery -A core worker -l info
volumes:
- ./app:/usr/src/app
env_file:
- ./.env.dev
depends_on:
- redis
celery-beat:
build:
context: ./app
dockerfile: Dockerfile.dev
command: celery -A core beat -l info
volumes:
- ./app:/usr/src/app
env_file:
- ./.env.dev
depends_on:
- redis
Try using http://localhost:8000/ (don't forget "/" at the end) and it should works