This question is related to a deadlock that happens (at times as described below) when importing an osgi service using spring-dm.
I have recently come across this issue in Spring-DM's ServiceDynamicInterceptor class that leads to a dead lock situation. This happens when two applications contexts are loaded from two bundles. Assume that app ctx A exports an osgi service and app ctx B imports the osgi service exported from A. The problem happens when both bundles are started together. When app ctx A gets initialized, spring calls the listeners registered for the service synchronously during publishing the osgi service.. Assume that app ctx B gets initialized simultaneously and the osgi import gets executed. First it adds a listener for the osgi service to the bundle context and then goes to lookup the osgi service. Just as the listener gets added to bundle context, app ctx A picks it up and execute the listener. At this point, app ctx B, which acquires a lock, tries to look up the osgi service which is not yet published as the listener is being called which waits for the same lock to be released causing a deadlock. After spring bean creation timeout has reached in 5 minutes (default), app ctx B fails with BeanInitializationException caused by ServiceUnavailableException.
I have listed the code executed in the two threads.Note that both the lookup and the listener are defined in the same file - ServiceDynamicInterceptor.
App Ctx A is waiting for the lock at
145: synchronized (lock) {
146: servicePresent = (wrapper != null && wrapper.isServiceAlive());
147: }
App Ctx B is waiting for the osgi service to be published at
428: private Object lookupService() {
429: synchronized (lock) {
430: return (Object) retryTemplate.execute(retryCallback);
431: }
432: }
This happens at times As I said before, when both bundles are restarted together. However it does not happen every time for obvious reasons. My question is whether there is any way that this dead lock could be avoided?
Extracts from thread dump
App Ctx A
"SpringOsgiExtenderThread-90" Id=908 in BLOCKED on lock=java.lang.Object@5a9a333f
owned by pool-45-thread-1 Id=798
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor$Listener.serviceChanged(ServiceDynamicInterceptor.java:146)
at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:934)
at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:795)
at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:544)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4596)
at org.apache.felix.framework.Felix.registerService(Felix.java:3604)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
at org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean.registerService(OsgiServiceFactoryBean.java:310)
at org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean.registerService(OsgiServiceFactoryBean.java:279)
at org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean$Executor.registerService(OsgiServiceFactoryBean.java:95)
at org.springframework.osgi.service.exporter.support.internal.controller.ExporterController.registerService(ExporterController.java:40)
at org.springframework.osgi.service.dependency.internal.DefaultMandatoryDependencyManager.startExporter(DefaultMandatoryDependencyManager.java:320)
at org.springframework.osgi.service.dependency.internal.DefaultMandatoryDependencyManager.checkIfExporterShouldStart(DefaultMandatoryDependencyManager.java:261)
at org.springframework.osgi.service.dependency.internal.DefaultMandatoryDependencyManager.discoverDependentImporterFor(DefaultMandatoryDependencyManager.java:254)
- locked org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean@2ae6baf0
at org.springframework.osgi.service.dependency.internal.DefaultMandatoryDependencyManager.addServiceExporter(DefaultMandatoryDependencyManager.java:187)
at org.springframework.osgi.service.dependency.internal.MandatoryDependencyBeanPostProcessor.postProcessAfterInitialization(MandatoryDependencyBeanPostProcessor.java:46)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:407)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1461)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
- locked java.util.concurrent.ConcurrentHashMap@25dfde15
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:589)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:69)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:355)
- locked java.lang.Object@c6d4820
at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:320)
at org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:132)
at java.lang.Thread.run(Thread.java:745)
App Ctx B
"pool-45-thread-1" Id=798 in TIMED_WAITING on lock=java.lang.Object@2b767b9b
at java.lang.Object.wait(Native Method)
at org.springframework.osgi.service.importer.support.internal.support.RetryTemplate.execute(RetryTemplate.java:106)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.lookupService(ServiceDynamicInterceptor.java:430)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:415)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.afterPropertiesSet(ServiceDynamicInterceptor.java:472)
at org.springframework.osgi.service.importer.support.OsgiServiceProxyFactoryBean.createProxy(OsgiServiceProxyFactoryBean.java:215)
at org.springframework.osgi.service.importer.support.AbstractServiceImporterProxyFactoryBean.getObject(AbstractServiceImporterProxyFactoryBean.java:86)
at org.springframework.osgi.service.importer.support.OsgiServiceProxyFactoryBean.getObject(OsgiServiceProxyFactoryBean.java:161)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
- locked java.util.concurrent.ConcurrentHashMap@1a8ffb40
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1442)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:248)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1109)
at org.apache.camel.spring.spi.ApplicationContextRegistry.lookup(ApplicationContextRegistry.java:43)
at org.apache.camel.impl.CompositeRegistry.lookup(CompositeRegistry.java:51)
at org.apache.camel.impl.PropertyPlaceholderDelegateRegistry.lookup(PropertyPlaceholderDelegateRegistry.java:62)
at org.apache.camel.util.CamelContextHelper.lookup(CamelContextHelper.java:119)
at org.apache.camel.model.RouteBuilderDefinition.createRouteBuilder(RouteBuilderDefinition.java:64)
at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.installRoutes(AbstractCamelContextFactoryBean.java:624)
at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.afterPropertiesSet(AbstractCamelContextFactoryBean.java:276)
at org.apache.camel.spring.CamelContextFactoryBean.afterPropertiesSet(CamelContextFactoryBean.java:235)
at org.apache.camel.osgi.CamelContextFactoryBean.afterPropertiesSet(CamelContextFactoryBean.java:64)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1514)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1452)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
- locked java.util.concurrent.ConcurrentHashMap@1a8ffb40
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:155)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:86)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:327)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.publishEvent(ServiceDynamicInterceptor.java:439)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.access$300(ServiceDynamicInterceptor.java:67)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor$EventSenderRetryTemplate.onMissingTarget(ServiceDynamicInterceptor.java:96)
at org.springframework.osgi.service.importer.support.internal.support.RetryTemplate.execute(RetryTemplate.java:95)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.lookupService(ServiceDynamicInterceptor.java:430)
- locked java.lang.Object@5a9a333f
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:415)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.afterPropertiesSet(ServiceDynamicInterceptor.java:472)
at org.springframework.osgi.service.importer.support.OsgiServiceProxyFactoryBean.createProxy(OsgiServiceProxyFactoryBean.java:215)
at org.springframework.osgi.service.importer.support.AbstractServiceImporterProxyFactoryBean.getObject(AbstractServiceImporterProxyFactoryBean.java:86)
at org.springframework.osgi.service.importer.support.OsgiServiceProxyFactoryBean.getObject(OsgiServiceProxyFactoryBean.java:161)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
- locked java.util.concurrent.ConcurrentHashMap@1a8ffb40
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1442)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:305)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:107)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
- locked java.util.concurrent.ConcurrentHashMap@1a8ffb40
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:607)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
- locked java.lang.Object@62b8345f
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.access$301(AbstractDelegatedExecutionApplicationContext.java:69)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$1.run(AbstractDelegatedExecutionApplicationContext.java:186)
at org.springframework.osgi.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.normalRefresh(AbstractDelegatedExecutionApplicationContext.java:182)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext$NoDependenciesWaitRefreshExecutor.refresh(AbstractDelegatedExecutionApplicationContext.java:89)
at org.springframework.osgi.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:175)
at com.xxx.bbb.core.extender.service.bbbManagedServiceFactory$1.run(BbbManagedServiceFactory.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
In the past I had similar problems with Spring-DM, the workaround that I did, was to track the service with a ServiceTracker via the BundleActivator. In some cases I also had to customize all the code that was relying on that service (eg: autowiring). It was ugly ... but it worked. After some experiments like this, I remember that I also used with success a FactoryBean that had the same approach ...