Monday, August 30, 2010

Retry Logic for Spring SimpleRemoteStatelessSessionProxyFactoryBean

This is one of the possible generic solution for a specific clustering/failover problem if we have a spring/ejb application deployed in Jboss server.

The problem will happen if we follow the pure distributed J2EE deploymnet having the web artifacts deployed on a set of web servers (Tomcat/JBoss etc) which remotely connects to a set of stateless session beans exposing business services. In this case session beans will be hosted on a cluster of JBoss servers.

The failover will work fine as long as the ejb cluster did not go for a complete shutdown. But in cases where the ejb application cluster was completely restarted for maintenance we will be forced to restart the web cluster as well since the client proxy no longer has the information on new cluster topology.

In the spring/ejb approache we will use the SimpleRemoteStatelessSessionProxyFactoryBean provided by spring to connect to the EJBs.

If we google we will find atleast 2 solutions

1. Use refreshHomeOnConnectFailure

The SimpleRemoteStatelessSessionProxyFactoryBean used for JNDI lookup has property “refreshHomeOnConnectFailure” which if sets true, Spring will refresh the Home by retrying.

The only catch here is the refresh will happen only for RMI Exception and its subclasses. But Jboss is throwing “java.lang.RuntimeException” exception and Spring will not do the the refresh.

Here is the JIRA for details

2. Use JBoss RetryInterceptor

With this interceptor, JBoss will refresh the proxy and it has to be specified in jboss.xml file.

This interceptor requires environment properties which can be set by using the method RetryInterceptor.setRetryEnv(env) or having the JNDI property file in the class path.

Since we are using spring, it is not possible to directly use setRetryEnv () method, instead we can extend the SimpleRemoteStatelessSessionProxyFactoryBean to have the retry lgic.

import java.lang.reflect.InvocationTargetException;
import java.rmi.RemoteException;

import javax.ejb.CreateException;
import javax.ejb.EJBObject;
import javax.naming.NamingException;

import org.aopalliance.intercept.MethodInvocation;
import org.springframework.ejb.access.SimpleRemoteStatelessSessionProxyFactoryBean;
import org.springframework.remoting.RemoteLookupFailureException;
import org.springframework.remoting.rmi.RmiClientInterceptorUtils;

* This is extended version of SimpleRemoteStatelessSessionProxyFactoryBean This
* class is extended to Retry the connetion after refreshing the ejbHome for a
* specfied number of times.
* This is only required in cases of entire cluster shutdown.
* For example: If there are nodeA,nodeB and nodeC in the cluster. If all these
* nodes are brought down then the client proxies do not have any knowledge on
* the cluster topology. In such a case, the only option is to refresh the home
* by a re-lookup and retry the operation.
* This can be used as spring replacement for the approach mentioned here
* @author Sujith Vellat
public class RetrySimpleRemoteStatelessSessionProxyFactoryBean extends
SimpleRemoteStatelessSessionProxyFactoryBean {

private int numberOfRetries = 2;

* This implementation "creates" a new EJB instance for each invocation. Can
* be overridden for custom invocation strategies.
* <p>
* Alternatively, override {@link #getSessionBeanInstance} and
* {@link #releaseSessionBeanInstance} to change EJB instance creation, for
* example to hold a single shared EJB component instance.
protected Object doInvoke(MethodInvocation invocation) throws Throwable {
return doInvoke(invocation, 0);

protected Object doInvoke(MethodInvocation invocation, int count)
throws Throwable {
Object ejb = null;
try {
ejb = getSessionBeanInstance();
if (count < numberOfRetries) {
return RmiClientInterceptorUtils.invokeRemoteMethod(invocation,
} else {
return null;
} catch (NamingException ex) {
throw new RemoteLookupFailureException(
"Failed to locate remote EJB [" + getJndiName() + "]", ex);
} catch (InvocationTargetException ex) {
Throwable targetEx = ex.getTargetException();
if (targetEx instanceof RemoteException) {
RemoteException rex = (RemoteException) targetEx;
throw RmiClientInterceptorUtils.convertRmiAccessException(
invocation.getMethod(), rex, isConnectFailure(rex),
} else if (targetEx instanceof CreateException) {
throw RmiClientInterceptorUtils.convertRmiAccessException(
invocation.getMethod(), targetEx,
"Could not create remote EJB [" + getJndiName() + "]");
} else if (targetEx instanceof RuntimeException) {
if ((targetEx.getCause() == null && "Unreachable?: Service unavailable."
|| (targetEx.getCause() != null && (targetEx.getCause() instanceof ConnectException || targetEx
.getCause().getCause() instanceof ConnectException))) {
try {
} catch (NamingException namingException) {
throw new RemoteLookupFailureException(
"Failed to locate remote EJB [" + getJndiName()
+ "]", namingException);
return doInvoke(invocation, ++count);
throw targetEx;
} finally {
if (ejb instanceof EJBObject) {
releaseSessionBeanInstance((EJBObject) ejb);

public int getNumberOfRetries() {
return numberOfRetries;

public void setNumberOfRetries(int numberOfRetries) {
this.numberOfRetries = numberOfRetries;


This approach gives us atleast a foolproof failover mechanism on JBoss server for the spring-ejb flavoured applications.

I would love to know if there are better approaches for the same problem.

Tuesday, July 27, 2010

More thoughts on Solution Architecture

My attempt to define solution architecture steps may have confused the readers and I started to receive more queries and I thought I will explain those in much more detail with examples.

NOTE: I will update this with an example to give more clarification as soon as I get some enough free time from my paid job :)

1) At what point will you decide on architecture pattern(style) e.g., Layers, SOA, REST, MVC ?
As we discussed earlier. There are tiers and different layers in each tier.
For example: At conceptual level itself you should be able to define the different tiers. A two tier, three tier or n-tier application.
The layers are identified at the Logical architecture level. For example in a n-tier app at the business tier I can have multiple layers based on various considerations (design approach, nfr , complexity, reusability etc.) like service layer, component layer, data access layer etc based on the concepts of separation of concerns / class-responsibilty-collaboration etc
SOA is a pattern/paradigm not a layer or tier. REST is a protocol to choose at the technical architecture level.

2) Where exactly are NFRs realised (In Conceptual or Logical architecture).
I would say NFRs are considered at all stages in architecture but the amount of impact of NFR in architecture differ at differnt phases. For example a performance requirement can reduce the number of tiers OR layers chosen for a system at the coceptual and logical levels.
The same performance requirement can introduce a caching layer at the time of logical/technical architecture phase. It can impact the technologies chosen at the tarchitecute phase. Since technical architecture being the pillar of a solution architecture this is where you get most impact and where you get your hands dirty :) Again architectures goes through different iterations to achieve its perfection, so when you define an architecture document for a project, put all your deep thoughts into it but keep it as a living document. Atleast thats what I found in my experience.

3) How exactly do you differenciate between Conceptual and Logical architecture.
I hope I answered this as part of frst question. Still to elaborate, the difference lies in the level of details. You also need to consdier that there are different audience for your architectue. This involves stake holders, other customer officials, business managers, sales team, testers and (the most important audience) developers who need to execute this. So we should be defining architecture at different detail levels for easiness of grasping the architecture for all type of audience. Many of the business audience/sales may not be much intersted in knowing all the layers in detail but may be interested in the conceptual architecture. I am not sure whether this will confuse you more but I would suggest you to compare conceptual-to-logical as a journey from highlevel design to lowlevel design (just to compare design is not architecture always).

Monday, July 26, 2010

Defining Solution Architecture

There are different types of architectures and architects in todays industry spanning from enterprise/solutions/system/technical/application/platform/data architectures/architects. (Architecture is now a very generic term) So the analysis of requirements to architecture differ for each of these cases.

Still let me try to explain the architecture at a solution architecture level as I understand enterprise architecture is still far from my reach :)

By understanding the business requirements and grouping them into modules/sub-systems at a high level we achieve the conceptual architecture. In normal scenarios we will be able to identify the different tiers in the system at this time.

These high level modules/subsystems are further analysed in more detail to get to a logical architecture and at this point we should be focusing on defining the various layers in each tier. This is achieved by identifying the business services, the reusable components that make up these services etc.

Now this logical architecture needs a technology mapping. Which technologies you choose for the each of the subsystems/tiers/layers are identified at this stage and this phase should form the pillar of our architecture. This is where the architecture reflects how the business requirements are transformed to technology solutions. This is where an architects breadth in technology knowledge comes handy and these chosen technologies are where a hands-on architect will increase his depth of knowledge. We also need to consider the non-functional requirements while choosing technologies and platforms. But please also note that non-functional requirements should be heavily prioritized as many of them can be compromised over the duration of a project. I like to call this phase as technical architecture (tarchitecture).

Once you have mapped your business requirements to technology, you need to define a physical view of how these technologies will be laid out and hence we define a Deployment architecture. The non-functional requirements like scalability, availability etc play a significant role while defining this. Whether our components should be co-located or distributed we decide at this stage assuming our tarchitecture supported both :)

So in short defining conceptual,logical,technical and deployment architectures are what i followed from the requirements to solution to delivery. There are more variations and additions and exceptions to these in specific cases which i skipped here and we can write a book to explain all those as we adapt many of our approaches based on each project :)

Monday, June 7, 2010

Before Beginning a Project

It has been a year since I wrote my first blog here. In this long period I was working on 2 software projects and I am almost near the final functional release of my first one. There is still some more work to be done but they are all what we call the final touches including minor bug fixes and performance improvements, enhancements based on wishlist that we kept on pending etc.

Straight to the point: Now a days many of the projects are carried out in a short time span, so what are some of the important steps that need to be put in place before beginning development of a software project.

  • Technically Foolproof Statement of Work (SOW)
  • Not many gets a chance to influence the SOW of projects that they work on. I strongly believe that any one who is going to play the architect role should be given a chance to review the SOW before its signed off if non-functional aspects like scalability and availability etc are detailed in the SOW documents. Some SOW writers go into extreme level in non-functional aspects like 99.9999 percent availability, infinite horizontal scalability, 100% test coverage etc with a motive of getting the project by sitting in fictional world. These are the statements which sets wrong expectation to any customer without giving them the entire details about the effort and cost to achieve the same. How to write a foolproof statement of work will need another blog of its entirety but I wanted to scribble some interesting thoughts I have on SOWs that may fail the project before it starts.

  • Proper Requirement Analysis:
  • I have seen organizations multi-tasking the resources for various type of day to day works in software industry. This has its pros/cons. Some times moving a core engineering team member into level 2 support or application testing team for some time helps the developer to understand the necessities of his counterparts and adapt his working style to contribute to the team better, but instead if developers are dragged into the requirement capturing phase we might end up in getting semi cooked design documents instead of requirement specifications. Instead of understanding the breadth and depth of requirements, the developer always tend to concentrate more on the technical challenges in achieving the same and misses out the important point of capturing the thoughts from the business team. So its important that organizations should not multi-task resources to this extreme and instead employ the right experienced resources at the time of requirement capturing.

  • Setting Up a Source Code Repository (SCR):

  • This is the place where we keep all the essential files required to build a product. It plays an important role in the software development life cycle and the significance is very high in recent projects where we have Continous Integration (CI) in place. SCR like Subversion enable the development team to work collaboratively and simultaneously on the code / document artifacts. These tools provide this concurrent collaboration feature with the help of powerful version control mechanism and conflict resolution inbuilt.

  • Defining the Standards:
  • However strong and experienced the team we build we should not forget that the team is a set of people with diversified thoughts and its necessary to ensure that all these thoughts are stream lined to achieve the success of a project as a team. Defining standards at all levels as much as possible ensure that this is achieved. Defining the set of tools that project should use, the development environment for all the different kinds of developers etc should be defined and documented upfront. Though I would say an architect will need to refine these things on the course of the project by receiving feedback from his team.

    General Developer Guidelines: This should be the first place of reference for any developer for any project specific documentation. In some situations this document may also act as the index for the other sub-documents.
    Coding Standards & Guidelines:

    Checklist: Code review checklists,

  • Setting up Continous Integration (CI):
  • “Continuous integration is a software development practice where members of a team integrate their work frequently—usually each person integrates at least daily—leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”—Martin Fowler

    There are so many servers supporting continous integration now and most of them are open source. I have used two of them Cruise Control and Hudson and found it exremely easy to set up and very helpful in the whole duration of projects.
    Regardless of what CI server you use, the CI server obtains the latest source code from SCR, locates all required dependencies, and builds the product independently from the previous build output.
    Well none of these CI servers will write the build scripts for us. So we need to ensure that we have a solid build system. (For eg: based on maven or ivy for a JEE project). Its very important that this is setup very early, atleast before the developers are kicked into the project.

  • Setting Up Bug Tracking System:
  • JIRA or bugzilla

  • Building an application framework

  • Maintaining a talent pool:
  • After working for some startups and consulting firms I find a talent pool playing a significant role in the success of any company. Maintaining a talent pool is going to be one of the major factors in the success of any organization. We save the time for team building. Having a nice team environment is not an easy one to achieve from the day one. Yes it takes time and effort. Apart from that the developers are already proven in the organization and hence you can guarantee that unhealthy competitions are avoided. Also when we have the organizational wide frameworks in place these developers will be well versed with the usage of the same.

    Friday, April 2, 2010

    Principles behind the Agile Manifesto

    1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

    2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

    3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

    4. Business people and developers must work together daily throughout the project.

    5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

    6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

    7. Working software is the primary measure of progress.

    8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

    9. Continuous attention to technical excellence and good design enhances agility.

    10. Simplicity—the art of maximizing the amount of work not done—is essential.

    11. The best architectures, requirements, and designs emerge from self-organizing teams.

    12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

    Highsmith (2001)