SOA Transaction timeout Settings with Error logs

This document explains the SOA transaction timeouts with an example error log. There are a quite a few timeouts involved in a SOA transaction.  The following rule should be followed when scheduling the transaction timeouts. BPEL SyncMaxWaitTime < BPEL EJB transaction timeout < Global Transaction Timeout (JTA timeout) <= XA transaction timeout < distributed lock timeout 1. SyncMaxWaitTime The syncMaxWaitTime is the timeout that a synchronous BPEL waits before timing out. The default value for syncMaxWaitTime property is 45. Let’s say you have a Caller BPEL Which is a synchronous service, invoking a calle bpel process which is also a synchronous process. If the calle bpel process takes morethan 45 seconds to send the response, the calle bpel process will time out with the below error.  <summary>request-response conversation has timed out for component = default/callebpelproject!1.0*soa_e05885cf-664c-4420-8551-180baacc39bd/syncBPELProcess, flowId = 90179</summary> Steps to configure SyncMaxWaitTime from EM console: a)      Login to EM. b)      Right click on SOA-infra à SOA Administrationà BPEL Properties. c)      Click on more Bpel Properties. d)      Scroll down for SyncMaxWaitTime property. Here you can alter the default value. 2. BPEL/BPM EJB Transaction Timeouts EJB transaction timeout is the timeout used by SOA Application. It terminates the transaction when it exceeds the timeout value. BPEL cannot resume after this timeout. It will rollback. No actions can be performed after this timeout in BPEL. These timeouts overrides the global transaction timeout specified by JTA. There are few ejb’s that can be modified to increase the ejb transaction timeout. Below is the list of EJB’s that needs to be modified if you decide to change the EJB transaction timeout. You can access them by using below steps. a)        Log into the Weblogic Console b)      Go to Deployments. c)       Expand soa-infra d)      EJBs:                 BPELActivityManagerBean                 BPELDeliveryBean                 BPELDispatcherBean                 BPELEngineBean                 BPELFinderBean                 BPELInstanceManagerBean                 BPELProcessManagerBean                 BPELSensorValuesBean                 BPELServerManagerBean  Below is the example of the EJB timeout exception. Transaction Rolledback.: weblogic.transaction.internal.TimedOutException: Transaction timed out after 300 seconds BEA1-1595060FF83E9E472B1E                 at weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransactionImpl.java:1842)                 at weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTransactions(ServerTransactionManagerImpl.java:1724)                 at org.glassfish.transaction.TransactionManagerImplCommon.wakeUp(TransactionManagerImplCommon.java:1216)                 at weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1636)                 at weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:20)                 at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:304)                 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) 3. JTA Timeout The Java Transaction API (JTA) timeout is set at the weblogic domain level. This is the global transaction timeout that happens for XA transactions. I have achieved the below error by calling a plsql package which executes a DBMS_LOCK.Sleep procudeure with input as greater than my JTA timeout. To Configure JTA Timeout- a)       Log into the Weblogic Console b)      Go to JTA. c)       Look for Timeout Seconds property. <bpelFault><faultType>0</faultType><runtimeFault xmlns="http://schemas.oracle.com/bpel/extension"><part name="summary"><summary>Global retry rollback fault thrown. The current JTA transaction is aborting due to an user rollback fault being thrown. The upstream component should retry in a new JTA transaction upon catching this fault. This exception was caused by a global retry fault being thrown from downstream component. The user had directed the BPEL engine to roll back the current JTA transaction and retry within new JTA transactions for the specified number of times and retry interval. There is no action recommended. </summary></part><part name="detail"><detail>ORABPEL-02198 Global retry rollback fault thrown. The current JTA transaction is aborting due to an user rollback fault being thrown. The upstream component should retry in a new JTA transaction upon catching this fault. This exception was caused by a global retry fault being thrown from downstream component. The user had directed the BPEL engine to roll back the current JTA transaction and retry within new JTA transactions for the specified number of times and retry interval. There is no action recommended.             at com.collaxa.cube.ws.WSInvocationManager.invoke(WSInvocationManager.java:343)             at com.collaxa.cube.engine.ext.common.InvokeHandler.__invoke(InvokeHandler.java:1179)             at com.collaxa.cube.engine.ext.common.InvokeHandler.handleNormalInvoke(InvokeHandler.java:638)             at com.collaxa.cube.engine.ext.common.InvokeHandler.handle(InvokeHandler.java:143)             at com.collaxa.cube.engine.ext.bpel.common.wmp.BPELInvokeWMP.__executeStatements(BPELInvokeWMP.java:74)             at com.collaxa.cube.engine.ext.bpel.common.wmp.BaseBPELActivityWMP.perform(BaseBPELActivityWMP.java:188)             at com.collaxa.cube.engine.CubeEngine.performActivity(CubeEngine.java:2864)             at com.collaxa.cube.engine.CubeEngine._handleWorkItem(CubeEngine.java:1260)             at com.collaxa.cube.engine.CubeEngine.handleWorkItem(CubeEngine.java:1155)             at com.collaxa.cube.engine.dispatch.message.instance.PerformMessageHandler.handleLocal(PerformMessageHandler.java:102)             at com.collaxa.cube.engine.dispatch.DispatchHelper.handleLocalMessage(DispatchHelper.java:308)             at com.collaxa.cube.engine.dispatch.DispatchHelper.sendMemory(DispatchHelper.java:387)             at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4855)             at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4779)             at com.collaxa.cube.engine.CubeEngine._createAndInvoke(CubeEngine.java:747)             at com.collaxa.cube.engine.CubeEngine.createAndInvoke(CubeEngine.java:586)             at com.collaxa.cube.engine.delivery.DeliveryService.handleInvoke(DeliveryService.java:634)             at com.collaxa.cube.engine.ejb.impl.CubeDeliveryBean.handleInvoke(CubeDeliveryBean.java:343)             at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Unknown Source)             at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:33)             at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.handleInvoke(Unknown Source)             at com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessageHandler.handle(InvokeInstanceMessageHandler.java:52)             at com.collaxa.cube.engine.dispatch.DispatchHelper.handleMessage(DispatchHelper.java:153)             at com.collaxa.cube.engine.dispatch.BaseDispatchTask.process(BaseDispatchTask.java:132)             at com.collaxa.cube.engine.dispatch.BaseDispatchTask.run(BaseDispatchTask.java:90)             at com.collaxa.cube.engine.dispatch.WMExecutor$W.run(WMExecutor.java:239)             at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)             at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)             at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) Caused by: oracle.fabric.common.FabricInvocationException: BINDING.JCA-12563 Exception occurred when binding was invoked. Exception occurred during invocation of JCA binding: "JCA Binding execute of Reference operation 'DB_Timeout' failed due to: Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml). ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.             at oracle.integration.platform.blocks.adapter.fw.jca.cci.EndpointInteractionException.getFabricInvocationException(EndpointInteractionException.java:83)             at oracle.integration.platform.blocks.adapter.AdapterReference.getFabricInvocationException(AdapterReference.java:426)             at oracle.integration.platform.blocks.adapter.AdapterReference.request(AdapterReference.java:271)             at oracle.integration.platform.blocks.mesh.SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:151)             at oracle.integration.platform.blocks.mesh.MessageRouter.request(MessageRouter.java:213)             at oracle.integration.platform.blocks.mesh.MeshImpl.request(MeshImpl.java:275)             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)             at java.lang.reflect.Method.invoke(Method.java:606)             at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)             at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)             at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)             at oracle.integration.platform.metrics.PhaseEventAspect.invoke(PhaseEventAspect.java:71)             at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)             at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)             at com.sun.proxy.$Proxy309.request(Unknown Source)             at oracle.fabric.CubeServiceEngine.requestToMesh(CubeServiceEngine.java:990)             at com.collaxa.cube.ws.WSInvocationManager.invoke(WSInvocationManager.java:244)             ... 28 more Caused by: BINDING.JCA-12563 Exception occurred when binding was invoked. Exception occurred during invocation of JCA binding: "JCA Binding execute of Reference operation 'DB_Timeout' failed due to: Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at "SYS.DBMS_LOCK", line 205 ORA-06512: at "PLSQL package", line 6 ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml). ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.executeJcaInteraction(JCAInteractionInvoker.java:569)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.invokeJcaReference(JCAInteractionInvoker.java:724)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.invokeSyncJcaReference(JCAInteractionInvoker.java:679)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAEndpointInteraction.performSynchronousInteraction(JCAEndpointInteraction.java:609)             at oracle.integration.platform.blocks.adapter.AdapterReference.request(AdapterReference.java:220)             ... 44 more Caused by: BINDING.JCA-11811 Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml).             at oracle.tip.adapter.db.exceptions.DBResourceException.createXARetriableException(DBResourceException.java:678)             at oracle.tip.adapter.db.exceptions.DBResourceException.createEISException(DBResourceException.java:650)             at oracle.tip.adapter.db.sp.SPUtil.createResourceException(SPUtil.java:186)             at oracle.tip.adapter.db.sp.AbstractStoredProcedure.execute(AbstractStoredProcedure.java:149)             at oracle.tip.adapter.db.sp.SPInteraction.executeStoredProcedure(SPInteraction.java:146)             at oracle.tip.adapter.db.DBInteraction.executeStoredProcedure(DBInteraction.java:1302)             at oracle.tip.adapter.db.DBInteraction.execute(DBInteraction.java:307)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.executeJcaInteraction(JCAInteractionInvoker.java:415)             ... 48 more Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at "SYS.DBMS_LOCK", line 205 ORA-06512: at "PLSQL PACKAGE", line 6 ORA-06512: at line 1             at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:466)             at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:407)             at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1113)             at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:546)             at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)             at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:603)             at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:228)             at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:49)             at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1000)             at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1316)             at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:5010)             at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:5195)             at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:10814)             at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1546)             at weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:105)             at oracle.tip.adapter.db.sp.AbstractStoredProcedure.execute(AbstractStoredProcedure.java:141)             ... 52 more </detail></part><part name="code"><code>com.collaxa.cube.engine.EngineException</code></part></runtimeFault></bpelFault>  4. XA Transaction (Global) Timeout. The final timeout configurable within the WebLogic console is the JDBC XA Transaction Timeout (or XA timeout, for short).  This is configurable on all of the available datasources associated with the transaction, which includes application specific datasources. For example, here is an exception from a JDBC connection timeout. I got this from oracle documentation. Caused by: weblogic.transaction.TimedOutException: Transaction timed out after 341 seconds BEA1-1CB4156557436868781B at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1721) ... 72 more at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1629) at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499) at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483) at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:527) at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513) at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:506) at br.com.multiplusfidelidade.submeterfulfillmentisbpm.FaultHandler.saveFault(FaultHandler.java:90) at br.com.multiplusfidelidade.submeterfulfillmentisbpm.FaultHandler.handleFault(FaultHandler.java:73) at com.collaxa.cube.engine.fp.RecoveryActionJava.execute(RecoveryActionJava.java:74) at com.collaxa.cube.engine.fp.BPELRecoverFault.recover(BPELRecoverFault.java:87) at oracle.fabric.CubeServiceEngine.recoverFault(CubeServiceEngine.java:1954) at oracle.integration.platform.faultpolicy.RecoverFault.recoverAndChain(RecoverFault.java:161) at oracle.integration.platform.faultpolicy.RecoverFault.resolveAndRecover(RecoverFault.java:124) at oracle.integration.platform.faultpolicy.FaultRecoveryManagerImpl.resolveAndRecover(FaultRecoveryManagerImpl.java:121) at com.collaxa.cube.engine.ext.common.FaultPolicyHandler.resolveAndRecover(FaultPolicyHandler.java:90) at com.collaxa.cube.engine.ext.common.InvokeHandler.handleException(InvokeHandler.java:697) ... In the above exception you can see that the engine attempted to perform a fault recovery and probably entered a custom class to do the recovery which ended up making a JDBC call that timed out. In some scenarios, you might need to disable the XA Transaction Timeout (to disable it, set it to 0). Such is the case when you face a "java.sql.SQLException: XA error: XAResource.XAER_NOTA start() failed on resource '<resource-name>': XAER_NOTA : The XID is not valid."  In order to check jdbc xa timeout: Login to Admin Console Expand Services Click on DataSources associated (configurable on all of the available datasources associated with the transaction, which includes application specific datasources) Configuration --> Transaction Check XA Transaction Timeout 5. Distributed Lock Timeout This timeout is set at the database level. The default timeout is 60 seconds.

References: Oracle Documentation