upgradeartifactoryderby

Upgrading Artifactory 6.23.42 to 7.41.14 failing derby db migration v60


Attempting to upgrade from Artifactory 6.23.42 to 7.41.14 (both Pro). The 7.41.14 is the last version that supports jdk 11 and there is unspecific warning about JDK 17 support on CentOS 7.

Note that JDK 17 and Tomcat 9.x are not supported on all Centos 7.x versions.

My environment:

What I did:

Results:

The db migration completes a bunch of steps but dies with: Failed to execute query: 'ALTER TABLE' cannot be performed on 'REPLICATION_ERRORS' because it does not exist.

The artifactory-service.log shows a failed db migration:

2023-04-06T17:18:15.512Z [jfrt ] [INFO ] [af075423d2f70fd8] [o.j.c.AbstractDbConverter:51  ] [Catalina-utility-2  ] - Converter `v59` is done with status `SKIPPED`.
2023-04-06T17:18:15.531Z [jfrt ] [INFO ] [af075423d2f70fd8] [o.j.c.AbstractDbConverter:36  ] [Catalina-utility-2  ] - Starting converter v60 with description: "Add column `event_category` to table `replication_errors`."
2023-04-06T17:18:15.532Z [jfrt ] [INFO ] [af075423d2f70fd8] [j.c.u.PreDefinedConverters:398] [Catalina-utility-2  ] - Searching for default DB conversion template at /templates/default/add_column.tpl
2023-04-06T17:18:15.534Z [jfrt ] [INFO ] [af075423d2f70fd8] [j.c.u.PreDefinedConverters:404] [Catalina-utility-2  ] - Searching for concrete DB (derby) conversion template at /templates/derby/add_column.tpl
2023-04-06T17:18:15.539Z [jfrt ] [WARN ] [af075423d2f70fd8] [o.j.s.u.DbStatementUtils:62   ] [Catalina-utility-2  ] - Failed to execute query: 'ALTER TABLE' cannot be performed on 'REPLICATION_ERRORS' because it does not exist.:
ALTER TABLE replication_errors ADD COLUMN event_category VARCHAR(225)
2023-04-06T17:18:15.540Z [jfrt ] [INFO ] [af075423d2f70fd8] [o.j.c.AbstractDbConverter:51  ] [Catalina-utility-2  ] - Converter `v60` is done with status `FAILED`.
2023-04-06T17:18:15.552Z [jfrt ] [ERROR] [af075423d2f70fd8] [d.c.m.ConverterManagerImpl:278] [Catalina-utility-2  ] - Conversion failed. You should analyze the error and retry launching Artifactory. Error is: Failed to convert v60 : Failed to convert v60
2023-04-06T17:18:15.557Z [jfrt ] [ERROR] [af075423d2f70fd8] [tifactoryHomeConfigListener:55] [Catalina-utility-2  ] - Failed initializing Home. Caught exception:
java.lang.IllegalStateException: Failed to convert v60 : Failed to convert v60
       at org.artifactory.storage.db.converter.markers.ConverterManagerImpl.handleException(ConverterManagerImpl.java:281)
       at org.artifactory.storage.db.converter.markers.ConverterManagerImpl.serviceConvert(ConverterManagerImpl.java:238)
       at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
       at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
       at org.artifactory.storage.db.converter.markers.ConverterManagerImpl.convertDatabase(ConverterManagerImpl.java:156)
       at org.artifactory.lifecycle.storage.db.init.PostDbSchemaInitializationManager.convertAndInit(PostDbSchemaInitializationManager.java:61)
       at org.artifactory.lifecycle.storage.db.init.HaInitLock.runInsideInitLockIfNeeded(HaInitLock.java:110)
       at org.artifactory.lifecycle.storage.db.init.PostDbSchemaInitializationManager.init(PostDbSchemaInitializationManager.java:50)
       at org.artifactory.lifecycle.webapp.servlet.BasicConfigurationManager.initArtifactoryInstallation(BasicConfigurationManager.java:154)
       at org.artifactory.lifecycle.webapp.servlet.BasicConfigurationManager.initialize(BasicConfigurationManager.java:126)
       at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.initBasicConfigManager(ArtifactoryHomeConfigListener.java:61)
       at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.contextInitialized(ArtifactoryHomeConfigListener.java:53)
       at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
       at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
       at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
       at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
       at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
       at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
       at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690)
       at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.jfrog.converter.DbConverterRuntimeException: Failed to convert v60
       at org.artifactory.storage.db.DatabaseConverterAdapter.convert(DatabaseConverterAdapter.java:49)
       at org.artifactory.storage.db.converter.markers.ConverterManagerImpl.serviceConvert(ConverterManagerImpl.java:234)
       ... 25 common frames omitted
Caused by: org.jfrog.converter.DbConverterExecutionException: Failed to convert v60
       at org.jfrog.converter.util.DbConverterUtil.executeConverters(DbConverterUtil.java:79)
       at org.jfrog.converter.util.DbConverterUtil.calculateAndExecute(DbConverterUtil.java:114)
       at org.artifactory.storage.db.DatabaseConverterAdapter.convert(DatabaseConverterAdapter.java:44)
       ... 26 common frames omitted
Caused by: org.jfrog.converter.DbConverterExecutionException: Failed to convert v60
       at org.jfrog.converter.predefined.PreDefinedDbConverter.executeConverter(PreDefinedDbConverter.java:37)
       at org.jfrog.converter.AbstractDbConverter.convert(AbstractDbConverter.java:39)
       at org.jfrog.converter.util.DbConverterUtil.executeConverters(DbConverterUtil.java:75)
       ... 28 common frames omitted
Caused by: java.sql.SQLSyntaxErrorException: 'ALTER TABLE' cannot be performed on 'REPLICATION_ERRORS' because it does not exist.
       at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
       at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
       at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
       at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
       at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
       at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
       at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
       at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source)
       at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source)
       at com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:120)
       at com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
       at org.jfrog.storage.dbtype.DefaultDbTypeHelper.executeUpdateQuery(DefaultDbTypeHelper.java:68)
       at org.jfrog.storage.util.DbStatementUtils.executeSqlStream(DbStatementUtils.java:57)
       at org.jfrog.converter.predefined.PreDefinedDbConverter.executeConverter(PreDefinedDbConverter.java:35)
       ... 30 common frames omitted
Caused by: org.apache.derby.iapi.error.StandardException: 'ALTER TABLE' cannot be performed on 'REPLICATION_ERRORS' because it does not exist.
       at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
       at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
       at org.apache.derby.impl.sql.compile.DDLStatementNode.justGetDescriptor(Unknown Source)
       at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(Unknown Source)
       at org.apache.derby.impl.sql.compile.DDLStatementNode.getTableDescriptor(Unknown Source)
       at org.apache.derby.impl.sql.compile.AlterTableNode.bindStatement(Unknown Source)
       at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
       at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
       at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
       ... 38 common frames omitted
2023-04-06T17:18:15.577Z [jfrt ] [ERROR] [af075423d2f70fd8] [actoryContextConfigListener:90] [Catalina-utility-2  ] - Failed initializing Artifactory context: Artifactory home not initialized.

Solution

  • As it seems during the upgrade process,Artifactory had an issue creating the 'REPLICATION_ERRORS' table, if this behavior persists after trying to restart the service, it is possible to create this table manually, and it should help resolve this error.

    As it require to perform manual changes to the DB, it is recommended to backup the DB before making the changes.

    You may use the following workaround the create the missing table and its corresponding indexes:

    1. Stop Artifacory
    2. Connect to your Derby DB by following these instructions (for Artifactory 7.x)
    3. Run the following SQL queries in the Derby database:
    CREATE TABLE replication_errors
    (
        error_id         BIGINT        NOT NULL,
        first_error_time BIGINT        NOT NULL,
        last_error_time  BIGINT        NOT NULL,
        error_count      SMALLINT      NOT NULL,
        error_message    VARCHAR(4000) NOT NULL,
        replication_key  VARCHAR(255)  NOT NULL,
        task_time        BIGINT        NOT NULL,
        task_type        SMALLINT      NOT NULL,
        task_path        VARCHAR(1344) NOT NULL,
        CONSTRAINT replication_errors_pk PRIMARY KEY (error_id)
    );
    
    1. After creating the table, run this command to create its corresponding index:

    CREATE INDEX replication_errors_rep_key_idx ON replication_errors (replication_key);

    1. Update the "schema_change_log" table so it can rerun the skipped conversions:

    UPDATE schema_change_log SET status='FAILED' WHERE version='v60';

    1. Once done with the changes, disconnect from the Derby DB and start Artifactory.