SOA Transaction timeout Settings with Error logs

This document explains the SOA transaction timeouts with an example error log. There are a quite a few timeouts involved in a SOA transaction.  The following rule should be followed when scheduling the transaction timeouts. BPEL SyncMaxWaitTime < BPEL EJB transaction timeout < Global Transaction Timeout (JTA timeout) <= XA transaction timeout < distributed lock timeout 1. SyncMaxWaitTime The syncMaxWaitTime is the timeout that a synchronous BPEL waits before timing out. The default value for syncMaxWaitTime property is 45. Let’s say you have a Caller BPEL Which is a synchronous service, invoking a calle bpel process which is also a synchronous process. If the calle bpel process takes morethan 45 seconds to send the response, the calle bpel process will time out with the below error.  <summary>request-response conversation has timed out for component = default/callebpelproject!1.0*soa_e05885cf-664c-4420-8551-180baacc39bd/syncBPELProcess, flowId = 90179</summary> Steps to configure SyncMaxWaitTime from EM console: a)      Login to EM. b)      Right click on SOA-infra à SOA Administrationà BPEL Properties. c)      Click on more Bpel Properties. d)      Scroll down for SyncMaxWaitTime property. Here you can alter the default value. 2. BPEL/BPM EJB Transaction Timeouts EJB transaction timeout is the timeout used by SOA Application. It terminates the transaction when it exceeds the timeout value. BPEL cannot resume after this timeout. It will rollback. No actions can be performed after this timeout in BPEL. These timeouts overrides the global transaction timeout specified by JTA. There are few ejb’s that can be modified to increase the ejb transaction timeout. Below is the list of EJB’s that needs to be modified if you decide to change the EJB transaction timeout. You can access them by using below steps. a)        Log into the Weblogic Console b)      Go to Deployments. c)       Expand soa-infra d)      EJBs:                 BPELActivityManagerBean                 BPELDeliveryBean                 BPELDispatcherBean                 BPELEngineBean                 BPELFinderBean                 BPELInstanceManagerBean                 BPELProcessManagerBean                 BPELSensorValuesBean                 BPELServerManagerBean  Below is the example of the EJB timeout exception. Transaction Rolledback.: weblogic.transaction.internal.TimedOutException: Transaction timed out after 300 seconds BEA1-1595060FF83E9E472B1E                 at weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransactionImpl.java:1842)                 at weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTransactions(ServerTransactionManagerImpl.java:1724)                 at org.glassfish.transaction.TransactionManagerImplCommon.wakeUp(TransactionManagerImplCommon.java:1216)                 at weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1636)                 at weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:20)                 at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:304)                 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) 3. JTA Timeout The Java Transaction API (JTA) timeout is set at the weblogic domain level. This is the global transaction timeout that happens for XA transactions. I have achieved the below error by calling a plsql package which executes a DBMS_LOCK.Sleep procudeure with input as greater than my JTA timeout. To Configure JTA Timeout- a)       Log into the Weblogic Console b)      Go to JTA. c)       Look for Timeout Seconds property. <bpelFault><faultType>0</faultType><runtimeFault xmlns="http://schemas.oracle.com/bpel/extension"><part name="summary"><summary>Global retry rollback fault thrown. The current JTA transaction is aborting due to an user rollback fault being thrown. The upstream component should retry in a new JTA transaction upon catching this fault. This exception was caused by a global retry fault being thrown from downstream component. The user had directed the BPEL engine to roll back the current JTA transaction and retry within new JTA transactions for the specified number of times and retry interval. There is no action recommended. </summary></part><part name="detail"><detail>ORABPEL-02198 Global retry rollback fault thrown. The current JTA transaction is aborting due to an user rollback fault being thrown. The upstream component should retry in a new JTA transaction upon catching this fault. This exception was caused by a global retry fault being thrown from downstream component. The user had directed the BPEL engine to roll back the current JTA transaction and retry within new JTA transactions for the specified number of times and retry interval. There is no action recommended.             at com.collaxa.cube.ws.WSInvocationManager.invoke(WSInvocationManager.java:343)             at com.collaxa.cube.engine.ext.common.InvokeHandler.__invoke(InvokeHandler.java:1179)             at com.collaxa.cube.engine.ext.common.InvokeHandler.handleNormalInvoke(InvokeHandler.java:638)             at com.collaxa.cube.engine.ext.common.InvokeHandler.handle(InvokeHandler.java:143)             at com.collaxa.cube.engine.ext.bpel.common.wmp.BPELInvokeWMP.__executeStatements(BPELInvokeWMP.java:74)             at com.collaxa.cube.engine.ext.bpel.common.wmp.BaseBPELActivityWMP.perform(BaseBPELActivityWMP.java:188)             at com.collaxa.cube.engine.CubeEngine.performActivity(CubeEngine.java:2864)             at com.collaxa.cube.engine.CubeEngine._handleWorkItem(CubeEngine.java:1260)             at com.collaxa.cube.engine.CubeEngine.handleWorkItem(CubeEngine.java:1155)             at com.collaxa.cube.engine.dispatch.message.instance.PerformMessageHandler.handleLocal(PerformMessageHandler.java:102)             at com.collaxa.cube.engine.dispatch.DispatchHelper.handleLocalMessage(DispatchHelper.java:308)             at com.collaxa.cube.engine.dispatch.DispatchHelper.sendMemory(DispatchHelper.java:387)             at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4855)             at com.collaxa.cube.engine.CubeEngine.endRequest(CubeEngine.java:4779)             at com.collaxa.cube.engine.CubeEngine._createAndInvoke(CubeEngine.java:747)             at com.collaxa.cube.engine.CubeEngine.createAndInvoke(CubeEngine.java:586)             at com.collaxa.cube.engine.delivery.DeliveryService.handleInvoke(DeliveryService.java:634)             at com.collaxa.cube.engine.ejb.impl.CubeDeliveryBean.handleInvoke(CubeDeliveryBean.java:343)             at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Unknown Source)             at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:33)             at com.collaxa.cube.engine.ejb.impl.bpel.BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.handleInvoke(Unknown Source)             at com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessageHandler.handle(InvokeInstanceMessageHandler.java:52)             at com.collaxa.cube.engine.dispatch.DispatchHelper.handleMessage(DispatchHelper.java:153)             at com.collaxa.cube.engine.dispatch.BaseDispatchTask.process(BaseDispatchTask.java:132)             at com.collaxa.cube.engine.dispatch.BaseDispatchTask.run(BaseDispatchTask.java:90)             at com.collaxa.cube.engine.dispatch.WMExecutor$W.run(WMExecutor.java:239)             at weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)             at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)             at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) Caused by: oracle.fabric.common.FabricInvocationException: BINDING.JCA-12563 Exception occurred when binding was invoked. Exception occurred during invocation of JCA binding: "JCA Binding execute of Reference operation 'DB_Timeout' failed due to: Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml). ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.             at oracle.integration.platform.blocks.adapter.fw.jca.cci.EndpointInteractionException.getFabricInvocationException(EndpointInteractionException.java:83)             at oracle.integration.platform.blocks.adapter.AdapterReference.getFabricInvocationException(AdapterReference.java:426)             at oracle.integration.platform.blocks.adapter.AdapterReference.request(AdapterReference.java:271)             at oracle.integration.platform.blocks.mesh.SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:151)             at oracle.integration.platform.blocks.mesh.MessageRouter.request(MessageRouter.java:213)             at oracle.integration.platform.blocks.mesh.MeshImpl.request(MeshImpl.java:275)             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)             at java.lang.reflect.Method.invoke(Method.java:606)             at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)             at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)             at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)             at oracle.integration.platform.metrics.PhaseEventAspect.invoke(PhaseEventAspect.java:71)             at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)             at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)             at com.sun.proxy.$Proxy309.request(Unknown Source)             at oracle.fabric.CubeServiceEngine.requestToMesh(CubeServiceEngine.java:990)             at com.collaxa.cube.ws.WSInvocationManager.invoke(WSInvocationManager.java:244)             ... 28 more Caused by: BINDING.JCA-12563 Exception occurred when binding was invoked. Exception occurred during invocation of JCA binding: "JCA Binding execute of Reference operation 'DB_Timeout' failed due to: Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at "SYS.DBMS_LOCK", line 205 ORA-06512: at "PLSQL package", line 6 ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml). ". The invoked JCA adapter raised a resource exception. Please examine the above error message carefully to determine a resolution.             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.executeJcaInteraction(JCAInteractionInvoker.java:569)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.invokeJcaReference(JCAInteractionInvoker.java:724)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.invokeSyncJcaReference(JCAInteractionInvoker.java:679)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAEndpointInteraction.performSynchronousInteraction(JCAEndpointInteraction.java:609)             at oracle.integration.platform.blocks.adapter.AdapterReference.request(AdapterReference.java:220)             ... 44 more Caused by: BINDING.JCA-11811 Stored procedure invocation error. Error while trying to prepare and execute the PLSQL API. An error occurred while preparing and executing the PLSQL API. Cause: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at line 1 Check to ensure that the API is defined in the database and that the parameters match the signature of the API.  This exception is considered retriable, likely due to a communication failure.  Because the global transaction is rolling back the invoke must be retried in a new transaction, restarting from the place of the last transaction commit.  To classify it as non-retriable instead add property nonRetriableErrorCodes with value "1013" to your deployment descriptor (i.e. weblogic-ra.xml).             at oracle.tip.adapter.db.exceptions.DBResourceException.createXARetriableException(DBResourceException.java:678)             at oracle.tip.adapter.db.exceptions.DBResourceException.createEISException(DBResourceException.java:650)             at oracle.tip.adapter.db.sp.SPUtil.createResourceException(SPUtil.java:186)             at oracle.tip.adapter.db.sp.AbstractStoredProcedure.execute(AbstractStoredProcedure.java:149)             at oracle.tip.adapter.db.sp.SPInteraction.executeStoredProcedure(SPInteraction.java:146)             at oracle.tip.adapter.db.DBInteraction.executeStoredProcedure(DBInteraction.java:1302)             at oracle.tip.adapter.db.DBInteraction.execute(DBInteraction.java:307)             at oracle.integration.platform.blocks.adapter.fw.jca.cci.JCAInteractionInvoker.executeJcaInteraction(JCAInteractionInvoker.java:415)             ... 48 more Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation ORA-06512: at "SYS.DBMS_LOCK", line 205 ORA-06512: at "PLSQL PACKAGE", line 6 ORA-06512: at line 1             at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:466)             at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:407)             at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1113)             at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:546)             at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)             at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:603)             at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:228)             at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:49)             at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1000)             at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1316)             at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:5010)             at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:5195)             at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:10814)             at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1546)             at weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:105)             at oracle.tip.adapter.db.sp.AbstractStoredProcedure.execute(AbstractStoredProcedure.java:141)             ... 52 more </detail></part><part name="code"><code>com.collaxa.cube.engine.EngineException</code></part></runtimeFault></bpelFault>  4. XA Transaction (Global) Timeout. The final timeout configurable within the WebLogic console is the JDBC XA Transaction Timeout (or XA timeout, for short).  This is configurable on all of the available datasources associated with the transaction, which includes application specific datasources. For example, here is an exception from a JDBC connection timeout. I got this from oracle documentation. Caused by: weblogic.transaction.TimedOutException: Transaction timed out after 341 seconds BEA1-1CB4156557436868781B at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1721) ... 72 more at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1629) at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499) at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483) at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:527) at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513) at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:506) at br.com.multiplusfidelidade.submeterfulfillmentisbpm.FaultHandler.saveFault(FaultHandler.java:90) at br.com.multiplusfidelidade.submeterfulfillmentisbpm.FaultHandler.handleFault(FaultHandler.java:73) at com.collaxa.cube.engine.fp.RecoveryActionJava.execute(RecoveryActionJava.java:74) at com.collaxa.cube.engine.fp.BPELRecoverFault.recover(BPELRecoverFault.java:87) at oracle.fabric.CubeServiceEngine.recoverFault(CubeServiceEngine.java:1954) at oracle.integration.platform.faultpolicy.RecoverFault.recoverAndChain(RecoverFault.java:161) at oracle.integration.platform.faultpolicy.RecoverFault.resolveAndRecover(RecoverFault.java:124) at oracle.integration.platform.faultpolicy.FaultRecoveryManagerImpl.resolveAndRecover(FaultRecoveryManagerImpl.java:121) at com.collaxa.cube.engine.ext.common.FaultPolicyHandler.resolveAndRecover(FaultPolicyHandler.java:90) at com.collaxa.cube.engine.ext.common.InvokeHandler.handleException(InvokeHandler.java:697) ... In the above exception you can see that the engine attempted to perform a fault recovery and probably entered a custom class to do the recovery which ended up making a JDBC call that timed out. In some scenarios, you might need to disable the XA Transaction Timeout (to disable it, set it to 0). Such is the case when you face a "java.sql.SQLException: XA error: XAResource.XAER_NOTA start() failed on resource '<resource-name>': XAER_NOTA : The XID is not valid."  In order to check jdbc xa timeout: Login to Admin Console Expand Services Click on DataSources associated (configurable on all of the available datasources associated with the transaction, which includes application specific datasources) Configuration --> Transaction Check XA Transaction Timeout 5. Distributed Lock Timeout This timeout is set at the database level. The default timeout is 60 seconds.

References: Oracle Documentation

XA and Non XA Transactions in SOA

There are two types of Transactions. 

- Non-XA - Local Transactions

- XA-Transactions

1) NON-XA Transaction - Non XA Transaction is a local transaction. It commits the transaction irrespective of the other adapters in the same business flow. Weblogic have provided an additional option to enable a support for global transactions for Non XA datasources too. This helps the local transaction resource manage to participate in global transaction. It uses the transaction manager of the global transaction resource as it doesnt have one of its own. It has three transaction options.

·         Logging Last Resource: With this option, the transaction branch in which the connection is used is processed as the last resource in the transaction and is processed as a local transaction. Commit records for two-phase commit (2PC) transactions are inserted in a table on the resource itself, and the result determines the success or failure of the prepare phase of the global transaction. This option offers some performance benefits and greater data safety than Emulate Two-Phase Commit, but it has some limitations.

·         Emulate Two-Phase Commit: With this option, the transaction branch in which the connection is used always returns success for the prepare phase of the transaction. It offers performance benefits, but also has risks to data in some failure conditions. Select this option only if your application can tolerate heuristic conditions.

·         One-Phase Commit: (selected by default) with this option, a connection from the data source can be the only participant in the global transaction and the transaction is completed using a one-phase commit optimization. If more than one resource participates in the transaction, an exception is thrown when the transaction manager calls XAResource.prepare on the 1PC resource.

2) XA Transactions – XA is a distributed transaction, also called a global transaction. Distributed transaction is a transaction that accesses and updates data on two or more networked resource WebLogic Server user Java Transaction API (JTA) to implement the distributed Transactions.

Global transaction achieves atomicity. In other words, each operation in the series must succeed for the entire transaction to be successful. If any operation in the transaction does not succeed, the entire transaction fails. At that point, any operations which have succeeded must be "rolled back" so that the end state matches what was in place before the transaction started. Global Transaction by default uses two-phase commit protocol to achieve atomicity.

Two Phase Commit: In the first phase, checks are made to see if the transaction ran without error for all the network resources involved in the transaction. If no errors, the results are committed for all the resources in the second phase. If there are errors in the first-phase check for any of the resources, the transaction is rolled back. This common transaction strategy is appropriately called the two-phase commit protocol.

Note: When we have a local Transaction (Non-XA) in between two global transaction(XA) calls, distributed transaction branch suspends, lets the same connection do a local transaction, and then resumes the branch later. The two-phase commit actions of distributed transaction do not affect the local transaction.

Java Transaction API(JTA):

The Java Transaction API (JTA) gives you the ability to perform distributed transactions, that is, an application can use the API to perform transactions on more than one data store in the network at the same time. But to do this efficiently, it helps to have another component operating in the application server: a transaction manager. A transaction manager helps to efficiently schedule and execute the potentially large number of transactions coming in through the application server.

I always wanted to know little deeper on how the JTA works. So here is some information about the methods that are invoked during global transaction.

XA transactions need a global transaction id and local transaction id(xid) for each XA resource. Each XA Resource is enlisted to XA Manager by start(xid) method. This method tells that XA Resource is being involved in the transaction(be ready for operations). After that, the first phase of the 2PC protocol is realized by calling prepare(xid) method. This method requests OK or ABORT vote from XA Resource. After receiving vote from each of XA Resource, XA Manager decides to execute a commit(xid) operation if all XA Resources send OK or decides to execute a rollback(xid) if an XA Resource sends ABORT. Finally, the end(xid) method is called for each of XA Resource telling that the transaction is completed. 

To learn more on SOA Transaction timeouts, please visit the link. http://pruthvisoa.blogspot.com/2018/08/soa-transaction-timeout-settings.html