springspring-bootspring-batch

Spring Batch Job is failing After upgrading to Spring batch5x version with other Spring libs with Java 17


We upgraded the Spring-boot-starter-parent from 2.7.0 to 3.0.1 along with JAVA 17 which in turn upgraded the Spring batch core from 4x to the latest 5x version. We also upgraded ojdbc9 to ojdbc11. We built the application and it is up but while running the batch jobs it is failing with the below error:

2023-10-19T18:24:19.330Z INFO 10 --- [qtp1552179791-302] c.o.f.c.a.c.ContentLcmPluginController   : Failed to deploy factory contents

org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)
        VALUES (?, ?, ?, ?)
]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction

        at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1539)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
        at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
        at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015)
        at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1020)
        at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:157)
        at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:168)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)

...

Caused by: java.sql.SQLException: ORA-08177: can't serialize access for this transaction

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:629)
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:563)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1230)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:298)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:511)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:162)
        at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1240)
        at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1819)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1471)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3760)
        at oracle.jdbc.driver.OraclePreparedStatement.executeLargeUpdate(OraclePreparedStatement.java:4061)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:4036)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:995)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
        at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
        ... 146 common frames omitted
Caused by: oracle.jdbc.OracleDatabaseException: ORA-08177: can't serialize access for this transaction

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:636)
        ... 163 common frames omitted

2023-10-19T18:24:19.338Z INFO 10 --- [qtp1552179791-302] com.oracle.faw.dp.lcm.util.LCMUtil       : Emitting metric for lcm plugin {PLUGIN_NAME=ContentServiceLcmPlugin, verb=content_importFactoryContentToService} execution
2023-10-19T18:24:19.338Z INFO 10 --- [qtp1552179791-302] c.o.f.c.a.c.ContentLcmPluginController   : Updating the lcm pugin execution status  for lcmVerb importFactoryContentToService  with values : pluginExecId 1,  status ERROR, errorCode 500  rawErrorMessage Install/associate factory contents failed. Please check logs.PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) 

{"level":"INFO","logger":"com.oracle.faw.cc.locks.LockManagerImpl","thread":"qtp1486553742-220","ts":1697791967796,"msg":"Released resource lock LockRequest(tenantId=oaxqa.cicdnr, resource=Resource(type=OAC, name=22.R1.P4), op=DEPLOY_FACTORY_CONTENT, owner=system) with result null","activityId":"21","correlation":"c4a0d325-0a59-44ce-9a76-5bcd4f2612e8","X-TENANT-ID":"oaxqa.cicdnr","raw_message":"Released resource lock {} with result {}"}
{"level":"ERROR","logger":"com.oracle.faw.content.api.controller.ContentLcmPluginController","thread":"qtp1486553742-220","ts":1697791967798,"msg":"Failed to deploy factory contents","activityId":"21","correlation":"c4a0d325-0a59-44ce-9a76-5bcd4f2612e8","X-TENANT-ID":"oaxqa.cicdnr","stack":"org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\n\n\tat org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1539)\n\tat org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)\n\tat org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)\n\tat org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015)\n\tat org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1020)\n\tat org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(JdbcJobInstanceDao.java:157)\n\tat org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:168)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)\n\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:391)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n\tat org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:207)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n\tat org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)\n\tat jdk.proxy2/jdk.proxy2.$Proxy117.createJobExecution(Unknown Source)\n\tat org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:145)\n\tat org.springframework.batch.core.launch.support.TaskExecutorJobLauncher.run(TaskExecutorJobLauncher.java:59)\n\tat com.oracle.faw.content.lcm.service.service.ContentJobLauncher.run(ContentJobLauncher.java:427)\n\tat com.oracle.faw.content.lcm.service.service.ContentJobLauncher.submitJob(ContentJobLauncher.java:414)\n\tat com.oracle.faw.content.lcm.service.service.ContentJobLauncher.factoryDeployJob(ContentJobLauncher.java:192)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:702)\n\tat com.oracle.faw.content.lcm.service.service.ContentJobLauncher$$SpringCGLIB$$0.factoryDeployJob(<generated>)\n\tat com.oracle.faw.content.lcm.service.service.ContentServiceImpl.deployFactoryContentAsync(ContentServiceImpl.java:130)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)\n\tat com.oracle.faw.cc.activity.aspects.ActivityAspect.processActivity(ActivityAspect.java:83)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:637)\n\tat org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:627)\n\tat org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:173)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:702)\n\tat com.oracle.faw.content.lcm.service.service.ContentServiceImpl$$SpringCGLIB$$0.deployFactoryContentAsync(<generated>)\n\tat com.oracle.faw.content.api.service.ContentLcmImportFactoryContentToService.deployFactoryContentAsync(ContentLcmImportFactoryContentToService.java:128)\n\tat com.oracle.faw.content.api.controller.ContentLcmPluginController.processImportFactoryContentToService(ContentLcmPluginController.java:819)\n\tat com.oracle.faw.content.api.controller.ContentLcmPluginController.importFactoryContentToService(ContentLcmPluginController.java:218)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)\n\tat org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:520)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)\n\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)\n\tat com.oracle.faw.content.api.filter.ResponseFilter.doFilter(ResponseFilter.java:42)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat com.oracle.faw.content.api.filter.RunningActivityFilter.doFilter(RunningActivityFilter.java:34)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat com.oracle.faw.content.api.filter.CorrelationHeaderFilter.doFilter(CorrelationHeaderFilter.java:57)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat com.oracle.faw.content.api.filter.AuthorizationFilter.doFilter(AuthorizationFilter.java:68)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1381)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:563)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)\n\tat org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)\n\tat java.base/java.lang.Thread.run(Thread.java:842)\nCaused by: java.sql.SQLException: ORA-08177: can't serialize access for this transaction\n\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:629)\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:563)\n\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1230)\n\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771)\n\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:298)\n\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:511)\n\tat oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:162)\n\tat oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1240)\n\tat oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1819)\n\tat oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1471)\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3760)\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeLargeUpdate(OraclePreparedStatement.java:4061)\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:4036)\n\tat oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:995)\n\tat com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)\n\tat com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)\n\tat org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)\n\tat org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)\n\t... 146 common frames omitted\nCaused by: oracle.jdbc.OracleDatabaseException: ORA-08177: can't serialize access for this transaction\n\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:636)\n\t... 163 common frames omitted\n","raw_message":"Failed to deploy factory contents"}
{"level":"ERROR","logger":"com.oracle.faw.dp.lcm.util.LCMUtil","thread":"qtp1486553742-220","ts":1697791967804,"msg":"Emitting metric for lcm plugin {PLUGIN_NAME=ContentServiceLcmPlugin, verb=content_importFactoryContentToService} execution","activityId":"21","correlation":"c4a0d325-0a59-44ce-9a76-5bcd4f2612e8","X-TENANT-ID":"oaxqa.cicdnr","raw_message":"Emitting metric for lcm plugin {} execution"}
{"level":"INFO","logger":"com.oracle.faw.content.api.controller.ContentLcmPluginController","thread":"qtp1486553742-220","ts":1697791967805,"msg":"Updating the lcm pugin execution status  for lcmVerb importFactoryContentToService  with values : pluginExecId 1,  status ERROR, errorCode 500  rawErrorMessage Install/associate factory contents failed. Please check logs.PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\norg.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\n, updatedBy system ","activityId":"21","correlation":"c4a0d325-0a59-44ce-9a76-5bcd4f2612e8","X-TENANT-ID":"oaxqa.cicdnr","raw_message":"Updating the lcm pugin execution status  for lcmVerb {}  with values : pluginExecId {},  status {}, errorCode {}  rawErrorMessage {}, updatedBy {} "}
{"level":"INFO","logger":"com.oracle.faw.dp.lcm.util.LCMUtil","thread":"qtp1486553742-220","ts":1697791967806,"msg":"In updateLCMEXECStatus sql >> update FAW_LCM_PLUGIN_EXEC set STATUS = ?, ERROR_CODE = ?, RAW_ERROR_MESSAGE= ?, UPDATED_BY = ?, UPDATED_AT = CURRENT_TIMESTAMP where ID = ? pluginExecId 1 status ERROR errorCode 500 rawErrorMessage Install/associate factory contents failed. Please check logs.PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\norg.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\n updatedBy system","activityId":"21","correlation":"c4a0d325-0a59-44ce-9a76-5bcd4f2612e8","X-TENANT-ID":"oaxqa.cicdnr","raw_message":"In updateLCMEXECStatus sql >> update FAW_LCM_PLUGIN_EXEC set STATUS = ?, ERROR_CODE = ?, RAW_ERROR_MESSAGE= ?, UPDATED_BY = ?, UPDATED_AT = CURRENT_TIMESTAMP where ID = ? pluginExecId 1 status ERROR errorCode 500 rawErrorMessage Install/associate factory contents failed. Please check logs.PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\norg.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION)\n\tVALUES (?, ?, ?, ?)\n]; SQL state [72000]; error code [8177]; ORA-08177: can't serialize access for this transaction\n updatedBy system"}

We migrated the table definition as below:

ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN TYPE_CD;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN KEY_NAME;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN STRING_VAL;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN DATE_VAL;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN LONG_VAL;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS DROP COLUMN DOUBLE_VAL;
ALTER TABLE BATCH_JOB_EXECUTION DROP COLUMN JOB_CONFIGURATION_LOCATION;
ALTER TABLE BATCH_STEP_EXECUTION DROP COLUMN START_TIME;
ALTER TABLE BATCH_STEP_EXECUTION ADD START_TIME TIMESTAMP DEFAULT NULL;
ALTER TABLE BATCH_STEP_EXECUTION ADD CREATE_TIME TIMESTAMP;
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS ADD PARAMETER_NAME VARCHAR(100);
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS ADD PARAMETER_TYPE VARCHAR(100);
ALTER TABLE BATCH_JOB_EXECUTION_PARAMS ADD PARAMETER_VALUE VARCHAR(2500);

I was thinking the issue may be with the below code where we were overriding to "DEFAULT_SERIALIZABLE". Updated it to "ISOLATION_SERIALIZABLE" as below but still no luck.

public JobRepository getJobRepository() throws Exception {
    JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
    factory.setDataSource(contentDataSource);
    factory.setTransactionManager(getTransactionManager());
    factory.setIsolationLevelForCreate("ISOLATION_SERIALIZABLE");
    factory.afterPropertiesSet();

    return factory.getObject();
}

Any help or input is appreciated. Thank you.


Solution

  • Finally found it. The problem is not related to Spring Batch code in particular but to the way Oracle has been creating tables since Version 11.2 - see here for details: https://oracle-base.com/articles/11g/segment-creation-on-demand-11gr2 The issue is in the segment creation and deferred part.

    To fix this:

    Thanks