dockerdocker-composemariadb

How do I correctly implement the tests requiring USAGE in the healthcheck.sh script from a healthcheck in my docker-compose.yml


I have an application that requires mariadb/mysql database. When putting both in the same docker-compose.yml, I run into an issue as the application is trying to access the database before it is fully started up. This is a common issue and is referenced multiple times across the internet. The solution is to implement a healthcheck on the mariadb container and have the application container depend_on the condition of service_health, aka the mariadb healthcheck passes.

MariaDB contains a healthcheck script location at /usr/local/bin/healthcheck.sh so I figured this would be pretty simple to implement. However, after searching I am unable to find solution, nor get anything but the --connect test to come back successful.

The following is my db portion of my docker-compose.yml file. I pulled out the web service as frankly it's irrelevant as it is just checking for the service_healthy condition of the db service.

---
version: '3.9'
services:
  db:
    image: mariadb:latest
    container_name: mariadb
    ports:
      - 3306:3306
    healthcheck:
      test: ["CMD", "/usr/local/bin/healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 10s
      timeout: 5s
      retries: 3
    volumes:
      - mariadb-vol:/var/lib/mysql
    networks:
      - proxy
    environment:
      - MYSQL_ROOT_PASSWORD=MysqlRootPassword
      - MYSQL_DATABASE=myappdb
      - MYSQL_USER=myapp
      - MYSQL_PASSWORD=MysqlRootPassword   
    restart: unless-stopped

networks:
  proxy:
    driver: bridge
    external: true
volumes:
  mariadb-vol:

It looks like the connect test passes, but the innodb_initialized fails.

I tried connecting to the docker container and running the /usr/local/bin/bin.healthcheck.sh script directly in bash but the --innodb_initialized still fails. I think that's related to the fact I'm logging into the container and it puts me as root. I'm not sure what user account the database is initialized as.


Solution

  • The innodb_initialized failure is an unfortunate lack of documentation now corrected with this page.

    The newer versions of that MariaDB container now include a healthcheck user with credentials populated in the .my-healthcheck.cnf file which will now work the the the compose file in the question.

    Recommended Configuration

    So update yaml configuration (replacing MYSQL_* with MARIADB_*) is:

    version: '3.9'
    services:
      db:
        image: mariadb:latest
        container_name: mariadb
        ports:
          - 3306:3306
        healthcheck:
          test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
          interval: 10s
          timeout: 5s
          retries: 3
        volumes:
          - mariadb-vol:/var/lib/mysql
        networks:
          - proxy
        environment:
          - MARIADB_ROOT_PASSWORD=MysqlRootPassword
          - MARIADB_DATABASE=myappdb
          - MARIADB_USER=myapp
          - MARIADB_PASSWORD=MysqlRootPassword
        restart: unless-stopped
    

    Prior to this change in the Docker Official Images, 10.4.30, 10.5.31 10.6.14, 10.9.7, 10.10.5, 10.11.4, 11.0.2, 11.1.1-rc is where the original question poster had troubles.

    In this case there wasn't a healthcheck user created or a .my-healthcheck.cnf file to provide credentials to perform the --innodb_initialized check. As such, the healthcheck.sh was run as root and, without a configuration file including a password, an empty root password was used.

    Reset user/password/privileges

    If;

    Remove the configuration file:

    docker run --rm -v mariadb-vol:/var/lib/mysql mariadb:latest rm /var/lib/mysql/.healthcheck.cnf
    

    Then start the container with environment MARIADB_AUTO_UPGRADE=1, and the users and configuration file will be re-created.

    Solution for older (obsolete) container versions

    One solution for those with an older initialized directory is to include the root password in a configuration file like and pass this is file in as a volume on /etc/mysql/conf.d/:

    healthcheck.cnf

    [mariadb-client]
    user=root
    password=MysqlRootPassword
    

    Horrible and very manual Configuration

    Another is to create healthcheck user and a .my-healthcheck.cnf file exactly how the updated entrypoint would have created the healthcheck user:

        CREATE USER healthcheck@'127.0.0.1' IDENTIFIED BY 'myVeryRandomPassword';
        CREATE USER healthcheck@'::1' IDENTIFIED BY 'myVeryRandomPassword';
        CREATE USER healthcheck@localhost IDENTIFIED BY 'myVeryRandomPassword';
        GRANT USAGE ON *.* TO healthcheck@'127.0.0.1';
        GRANT USAGE ON *.* TO healthcheck@'::1';
        GRANT USAGE ON *.* TO healthcheck@localhost;
    
    echo -e "[mariadb-client]\\nuser=healthcheck\\npassword=myVeryRandomPassword\\nprotocol=tcp\\n" |
       docker exec -i runningcontainer bash -c 'cat > /var/lib/mysql/.my-healthcheck.cnf'
    

    Passwordless Configuration

    If you are initializing and older volume and don't want to put a password into a configuration file, there is an alternative.

    The container environment variable MARIADB_MYSQL_LOCALHOST_USER=1 will create a mysql@localhost user with unix socket authentication. The default grants on this (USAGE), are enough to test innodb_initialized (see top of script). For this the compose file would look like where I have, for clarity only, emphasized an older version:

    version: '3.9'
    services:
      db:
        image: mariadb:10.4.29
        container_name: mariadb
        ports:
          - 3306:3306
        healthcheck:
          test: ["CMD", "healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
          interval: 10s
          timeout: 5s
          retries: 3
        volumes:
          - mariadb-vol:/var/lib/mysql
        networks:
          - proxy
        environment:
          - MARIADB_ROOT_PASSWORD=MysqlRootPassword
          - MARIADB_DATABASE=myappdb
          - MARIADB_USER=myapp
          - MARIADB_PASSWORD=MysqlRootPassword
          - MARIADB_MYSQL_LOCALHOST_USER=1   
        restart: unless-stopped
    

    Edit: Added MARIADB_AUTO_UPGRADE=1 will recreate config file and user if configuration file missing. Added more structure with headings.