When running mlflow with basic-auth get Worker exited with code 3.
I have my auth config set:
[mlflow]
default_permission = READ
database_uri = postgresql://un:pw@postgresdomain.com:5432/mlflowbackenddb
admin_username = customusername
admin_password = custompassword
authorization_function = mlflow.server.auth:authenticate_request_basic_auth
And my commands to run the server are:
export MLFLOW_TRACKING_URI=postgresql://un:pw@postgresdomain.com:5432/mlflowbackenddb
export MLFLOW_AUTH_CONFIG_PATH=/home/ec2-user/custom_auth_config.ini
mlflow server \
--backend-store-uri 'postgresql://un:pw@postgresdomain.com:5432/mlflowbackenddb' \
--app-name basic-auth
--default-artifact-root s3://bucket-address \
--artifacts-destination s3://bucket-address \
--host 0.0.0.0 --serve-artifacts --port 8080 &
When I run this, the server fails to launch. I did install postgres on the instance and check connection and it is able to connect.
The full error is verbose but here is a snippet:
2024/04/30 21:53:17 WARNING mlflow.server.auth: This feature is still experimental and may change in a future release without warning
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
2024/04/30 21:53:17 WARNING mlflow.server.auth: This feature is still experimental and may change in a future release without warning
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
2024/04/30 21:53:17 WARNING mlflow.server.auth: This feature is still experimental and may change in a future release without warning
2024/04/30 21:53:17 WARNING mlflow.server.auth: This feature is still experimental and may change in a future release without warning
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Worker (pid:7807) exited with code 3
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Worker (pid:7809) was sent SIGTERM!
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Worker (pid:7808) was sent SIGTERM!
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Worker (pid:7810) was sent SIGTERM!
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Shutting down: Master
[2024-04-30 21:53:17 +0000] [7806] [ERROR] Reason: Worker failed to boot.
I tried running these commands and following instructions from the official documentation at and from restacks guidance on postgres integration.
I changed postgres database engines from 16 to 15. I tried mysql database.
I CAN run the server if I take out basic-auth but then my server is not password protected.
I can confirm that when running without the --app-name tag, the server runs and the data is stored on the backend server. Just completely borks when trying to enable basic-auth.
There is an issue with your Database URI. Please change it to something like below
postgresql+psycopg2://un:pw@postgresdomain.com:5432/mlflowbackenddb
Basic auth is not working currently when you are using single database as basic auth has a seperate schema. Create a seperate db for basic auth(changed basic auth db in the config). With these changes your config file should look something like below.
[mlflow]
default_permission = READ
database_uri = postgresql+psycopg2://un:pw@postgresdomain.com:5432/mlflowauthdb
admin_username = customusername
admin_password = custompassword
authorization_function = mlflow.server.auth:authenticate_request_basic_auth
Run mlflow server with below command by updating db uri.
export MLFLOW_TRACKING_URI=postgresql+psycopg2://un:pw@postgresdomain.com:5432/mlflowbackenddb
export MLFLOW_AUTH_CONFIG_PATH=/home/ec2-user/custom_auth_config.ini
mlflow server \
--backend-store-uri $MLFLOW_TRACKING_URI \
--app-name basic-auth
--default-artifact-root s3://bucket-address \
--artifacts-destination s3://bucket-address \
--host 0.0.0.0 --serve-artifacts --port 8080