wso2wso2-api-managerwso2-das

Errors in API Analytics and API Manager


So I recently migrated my API Manager environment. I upgraded from 2.0.0 to 2.1.0 for both the API Manager and API Analytics. The API Analytics is on its own server. When I run the servers I get these errors

Analytics

    ERROR {org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent} -  Error in activating analytics data service: Unable to find class: piPublisher
        Serialization trace:
    columns (org.wso2.carbon.analytics.datasource.commons.AnalyticsSchema)
    schema (org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl$AnalyticsTableInfo) {org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent}
    com.esotericsoftware.kryo.KryoException: Unable to find class: piPublisher
    Serialization trace:
    columns (org.wso2.carbon.analytics.datasource.commons.AnalyticsSchema)
    schema (org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl$AnalyticsTableInfo)
            at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
            at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
            at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
            at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
            at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
            at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
            at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
            at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
            at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
            at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
            at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
            at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
            at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
            at org.wso2.carbon.analytics.datasource.core.util.GenericUtils.deserializeObject(GenericUtils.java:372)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.readTableInfo(AnalyticsDataServiceImpl.java:460)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.lookupTableInfo(AnalyticsDataServiceImpl.java:809)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.getTableSchemaFinal(AnalyticsDataServiceImpl.java:715)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.getTableSchema(AnalyticsDataServiceImpl.java:708)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.initIndexedTableStore(AnalyticsDataServiceImpl.java:210)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.<init>(AnalyticsDataServiceImpl.java:143)
            at org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent.activate(AnalyticsDataServiceComponent.java:69)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
            at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
            at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
            at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
            at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
            at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
            at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
            at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
            at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
            at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
            at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
            at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
            at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
            at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
            at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
            at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
            at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
            at org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:106)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:199)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:288)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
        at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451)
        at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:514)
        at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:219)
        at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:94)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
        at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
        at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
        at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: piPublisher cannot be found by kryo_2.24.0.wso2v1
        at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)
        at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)
        at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)

TID: [-1234] [] [2017-03-21 14:51:50,360] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} -  Dropping wrongly formatted event sent for -1234 {org.wso2.carbon.databridge.core.internal.queue.QueueWorker}
org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting loganalyzer:1.0.0 of event bundle with events 1
        at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:181)
        at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
        at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:73)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId loganalyzer:1.0.0 present in cache
        at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:166)

API Manager

UPDATE

I was able to fix these errors in the API Manager. See my comment below.

    TID: [-1] [] [2017-03-21 00:00:59,393] ERROR {org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter} -  Error when reporting timers {org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter}
java.sql.BatchUpdateException: Invalid object name 'METRIC_TIMER'.
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1178)
        at org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter.reportTimers(JDBCReporter.java:389)
        at org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter.report(JDBCReporter.java:200)
        at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
        at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
TID: [-1] [] [2017-03-21 00:01:59,391] ERROR {org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter} -  Error when reporting gauges {org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter}
java.sql.BatchUpdateException: Invalid object name 'METRIC_GAUGE'.
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1178)
        at org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter.reportGauges(JDBCReporter.java:221)
        at org.wso2.carbon.metrics.jdbc.reporter.JDBCReporter.report(JDBCReporter.java:188)
        at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
        at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

In the API Analytics is looks like there is a misspelled class or column name somewhere, but I can't find it in the configurations or database. I also tried to look in the components/plugins and was unable to really find anything. I'm guessing the errors in the API Manager are directly related to the errors in the Analytics server. I'm not sure how to trouble shoot this problem.


Solution

  • I fixed the error by deleting the ANX tables from the DB. My guess is that during migrate somethings did not go smoothly and was causing trouble for me.