artifactory

Artifactory "Access" Service not starting after 7.90.5 upgrade, DB migration fails (derby)


The release notes for 7.90.5 explain how the Access Service is now running in a separate JVM.

When using a derby DB the entire db is copied to $JFROG_HOME/artifactory/var/data/access/derby.

I manually set the db config for the access service, otherwise it tried to open the same db as the artifactory JVM process, which fails (multiple JVMs accessing one derby DB is not allowed). Here's the system.yaml:

shared:
    database:
        allowNonPostgresql: true
        type: derby
        driver: org.apache.derby.jdbc.EmbeddedDriver
        url: jdbc:derby:/opt/jfrog/artifactory/var/data/artifactory/derby;create=true
    user: artifactory
    extraJavaOpts: -Xss256k -Dkeystore.pkcs12.keyProtectionAlgorithm=PBEWithHmacSHA256AndAES_256
artifactory:
    port: 8081
    tomcat:
        connector:
            maxThreads: 200
            sendReasonPhrase: true
access:
    http:
        port: 8040
    database:
        type: derby
        driver: org.apache.derby.jdbc.EmbeddedDriver
        url: jdbc:derby:/opt/jfrog/artifactory/var/data/access/derby;create=true
    tomcat:
        connector:
            maxThreads: 50
            sendReasonPhrase: true

On startup, migrations are applied to the new database copy. This process fails.

Details from access-service.log:

2024-08-01T02:15:30.606Z [jfac ] [INFO ] [fa1259ae7c93acf5] [licationContextInitializer:165] [main                ] - Access (jfac) service initialization started. Version: 7.112.5 Revision: 81205900 PID: 14967 Home: /opt/jfrog/artifactory
2024-08-01T02:15:30.954Z [jfac ] [INFO ] [fa1259ae7c93acf5] [o.j.a.AccessApplication:55    ] [main                ] - Starting AccessApplication v7.112.5 using Java 17.0.10 on ip-172-31-10-237 with PID 14967 (/opt/jfrog/artifactory/app/access/tomcat/webapps/access/WEB-INF/lib/access-application-7.112.5.jar started by artifactory in /)
2024-08-01T02:15:30.955Z [jfac ] [INFO ] [fa1259ae7c93acf5] [o.j.a.AccessApplication:632   ] [main                ] - No active profile set, falling back to 1 default profile: "default"
2024-08-01T02:15:53.819Z [jfac ] [ERROR] [fa1259ae7c93acf5] [o.f.c.i.c.DbMigrate:294       ] [main                ] - Migration of schema "APP" to version 7.42 - Alter and remove index access tokens table failed! Changes successfully rolled back.
2024-08-01T02:15:53.839Z [jfac ] [ERROR] [fa1259ae7c93acf5] [.s.d.u.AccessJdbcHelperImpl:81] [main                ] - Could not initialize database:
org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:
Migration V7_42__Alter_and_remove_index_access_tokens_table.sql failed
----------------------------------------------------------------------
SQL State  : XSAI2
Error Code : 20000
Message    : The conglomerate (5,217) requested does not exist.
Location   : db/derby/migration/V7_42__Alter_and_remove_index_access_tokens_table.sql (/file:/opt/jfrog/artifactory/app/access/tomcat/webapps/access/WEB-INF/lib/access-server-core-7.112.5.jar!/db/derby/migration/V7_42__Alter_and_remove_index_access_tokens_table.sql)
Line       : 1
Statement  : DROP INDEX access_username_idx

IIUC the index that the migration attempts to drop doesn't exist.


Solution

  • It seems the derby db is corrupt, probably due to running out of disk space during the upgrade.

    I used ij to connect to it and ran select count(*) from TABLE for each table. That failed for two tables (it worked on the others):

    ij> select count(*) from ACCESS_PERMISSION_ACTION;
    ERROR XSAI2: The conglomerate (3,441) requested does not exist.
    
    ij> select count(*) from ACCESS_TOKENS;
    ERROR XSAI2: The conglomerate (5,217) requested does not exist.
    
    ij> select count(*) from BINARIES;
    1
    -----------
    252130
    
    1 row selected
    

    This error seems to indicate a corrupt database (Apache derby db Error ''The conglomerate (1.744) requested does not exist'').