Retry Logic for Spring SimpleRemoteStatelessSessionProxyFactoryBean

This is one of the possible generic solution for a specific clustering/failover problem if we have a spring/ejb application deployed in Jboss server.

The problem will happen if we follow the pure distributed J2EE deploymnet having the web artifacts deployed on a set of web servers (Tomcat/JBoss etc) which remotely connects to a set of stateless session beans exposing business services. In this case session beans will be hosted on a cluster of JBoss servers.

The failover will work fine as long as the ejb cluster did not go for a complete shutdown. But in cases where the ejb application cluster was completely restarted for maintenance we will be forced to restart the web cluster as well since the client proxy no longer has the information on new cluster topology.

In the spring/ejb approache we will use the SimpleRemoteStatelessSessionProxyFactoryBean provided by spring to connect to the EJBs.

If we google we will find atleast 2 solutions

1. Use refreshHomeOnConnectFailure

The SimpleRemoteStatelessSessionProxyFactoryBean used for JNDI lookup has property “refreshHomeOnConnectFailure” which if sets true, Spring will refresh the Home by retrying.

The only catch here is the refresh will happen only for RMI Exception and its subclasses. But Jboss is throwing “java.lang.RuntimeException” exception and Spring will not do the the refresh.

Here is the JIRA for details https://jira.springframework.org/browse/SPR-129

2. Use JBoss RetryInterceptor

With this interceptor, JBoss will refresh the proxy and it has to be specified in jboss.xml file.

http://community.jboss.org/wiki/RetryInterceptor

This interceptor requires environment properties which can be set by using the method RetryInterceptor.setRetryEnv(env) or having the JNDI property file in the class path.

Since we are using spring, it is not possible to directly use setRetryEnv () method, instead we can extend the SimpleRemoteStatelessSessionProxyFactoryBean to have the retry lgic.

import java.lang.reflect.InvocationTargetException;
import java.net.ConnectException;
import java.rmi.RemoteException;

import javax.ejb.CreateException;
import javax.ejb.EJBObject;
import javax.naming.NamingException;

import org.aopalliance.intercept.MethodInvocation;
import org.springframework.ejb.access.SimpleRemoteStatelessSessionProxyFactoryBean;
import org.springframework.remoting.RemoteLookupFailureException;
import org.springframework.remoting.rmi.RmiClientInterceptorUtils;

/**
 * This is extended version of SimpleRemoteStatelessSessionProxyFactoryBean This
 * class is extended to Retry the connetion after refreshing the ejbHome for a
 * specfied number of times.
 * 
 * This is only required in cases of entire cluster shutdown.
 * 
 * For example: If there are nodeA,nodeB and nodeC in the cluster. If all these
 * nodes are brought down then the client proxies do not have any knowledge on
 * the cluster topology. In such a case, the only option is to refresh the home
 * by a re-lookup and retry the operation.
 * 
 * This can be used as spring replacement for the approach mentioned here
 * http://community.jboss.org/wiki/RetryInterceptor
 * 
 * @author Sujith Vellat
 */
public class RetrySimpleRemoteStatelessSessionProxyFactoryBean extends
  SimpleRemoteStatelessSessionProxyFactoryBean {

 private int numberOfRetries = 2;

 /**
  * This implementation "creates" a new EJB instance for each invocation. Can
  * be overridden for custom invocation strategies.
  * <p>
  * Alternatively, override {@link #getSessionBeanInstance} and
  * {@link #releaseSessionBeanInstance} to change EJB instance creation, for
  * example to hold a single shared EJB component instance.
  */
 @Override
 protected Object doInvoke(MethodInvocation invocation) throws Throwable {
  return doInvoke(invocation, 0);
 }

 protected Object doInvoke(MethodInvocation invocation, int count)
   throws Throwable {
  Object ejb = null;
  try {
   ejb = getSessionBeanInstance();
   if (count < numberOfRetries) {
    return RmiClientInterceptorUtils.invokeRemoteMethod(invocation,
      ejb);
   } else {
    return null;
   }
  } catch (NamingException ex) {
   throw new RemoteLookupFailureException(
     "Failed to locate remote EJB [" + getJndiName() + "]", ex);
  } catch (InvocationTargetException ex) {
   Throwable targetEx = ex.getTargetException();
   if (targetEx instanceof RemoteException) {
    RemoteException rex = (RemoteException) targetEx;
    throw RmiClientInterceptorUtils.convertRmiAccessException(
      invocation.getMethod(), rex, isConnectFailure(rex),
      getJndiName());
   } else if (targetEx instanceof CreateException) {
    throw RmiClientInterceptorUtils.convertRmiAccessException(
      invocation.getMethod(), targetEx,
      "Could not create remote EJB [" + getJndiName() + "]");
   } else if (targetEx instanceof RuntimeException) {
    if ((targetEx.getCause() == null && "Unreachable?: Service unavailable."
      .equals(targetEx.getMessage()))
      || (targetEx.getCause() != null && (targetEx.getCause() instanceof ConnectException || targetEx
        .getCause().getCause() instanceof ConnectException))) {
     try {
      refreshHome();
     } catch (NamingException namingException) {
      throw new RemoteLookupFailureException(
        "Failed to locate remote EJB [" + getJndiName()
          + "]", namingException);
     }
     return doInvoke(invocation, ++count);
    }
   }
   throw targetEx;
  } finally {
   if (ejb instanceof EJBObject) {
    releaseSessionBeanInstance((EJBObject) ejb);
   }
  }
 }

 public int getNumberOfRetries() {
  return numberOfRetries;
 }

 public void setNumberOfRetries(int numberOfRetries) {
  this.numberOfRetries = numberOfRetries;
 }

}

This approach gives us atleast a foolproof failover mechanism on JBoss server for the spring-ejb flavoured applications.

I would love to know if there are better approaches for the same problem.