10 October 2008 - 15:47ESB transformations

ESBs is an essential enterprise computing capability which allows for mediating connections between various parties. Typical EBS usage consists of message producers sending messages to an ESB which transforms them and then forwards them to message consumers.

It is at the transformation stage that the thorniest issues arise with ESBs. As outlined in this presentation from Thoughworks there is a tendency to push business logic inside the ESB resulting in a very tight coupling between applications and the ESB (part of the business logic will be stored in applications, part of it in the ESB and the business logic in the ESB will need to be kept in sync with the business logic in the applications). Not a nice scenario and unfortunately the solution provided in the presentation (apparently you only need to use Guerilla SOA) doesn’t say much. The business logic which gets shoved into the ESB in the above example are the relationships between the various parties which are sending each other messages and the mess that comes out of it as these relationships change over time.

One example of ESB usage is a customer-facing web application which takes purchase orders from customers and sends messages to an inventory and shipping system, to the accounts receivables system and to the customer loyalty system. There are a few ways that this messages can be sent: 1) the web-application sends only one message to the ESB which then creates 3 more messages, one for the inventory system, one for the accounting system and one for the customer loyalty system (in which case the relationships between the web-app and the 3 systems are managed within the ESB); 2) the web-application creates 3 messages which are all sent independently to the 3 systems (in which case the relationship between the web-application and these 3 systems is managed by the web-app typically thru dedicated senders which are transforming the message according to the other party’s specs) or 3) the web-app publishes the message on the ESB which forwards it as-is to the receiving systems which are receiving the message as it has been created by the web-app and transform it according to their rules (in which case the receiving systems are managing the relationships with the web-application typically thru dedicated receiving points which are transforming the incoming messages).
So far I have only seen 2 or 3 used and it was used successfully. I have not seen 1 being used successfully, but rather I have seen ESBs simply delegated to the role of a dumb-pipe while the processing is done on the end-points. I think that the main reason for which the ESB cannot grow beyond the role of a dumb-pipe is that the transformations typically involve data, imagine that in the above scenario the IDs of the products on the web-application are different from the ones in the inventory system (a very common case BTW). In order to service this transformation you would need an up-to-date mapping between all relationships that it is managing. Sooner or later that ESB will turn into a giant vacuum cleaner that will need to suck every byte of data available in order to service the relationships stored in it.
Lesson #1: Managing relationships typically involves managing data.

Let’s go more into the gory details and think about how we would deal with stale data: let’s say that the datastore holding the mappings of product IDs from the webapp to the inventory systems is missing a product ID (another very common case). In the case where the relationships are maintained at the connection end-points (message producers or consumers) the operation team managing those end-points would have access to the exception (missing product ID mapping in this case) as well as to its solutions (create this mapping in your mapping store). How will the ESB deal with stale data? Well, the ESB will have to find the entity which can resolve this exception, package the exception and send it to that entity. Sooner or later the ESB gets becomes a spaghetti bowl.
Lesson #2: ESB has bad exception management capabilities. 

Pushing the transformation towards the end-points effectively couples the end-points because these endpoints must share the format (*) of the messages they exchange and, just as importantly, they need to share data. Sharing data is usually done by one party having a procedure to export its data and the other parties using this procedure for creating local replicas of that data and synch-ing them according to a calendar. In addition to that the parties replicating the data have procedure for overriding the data they receive and adding to it. As far as I have seen it works pretty well and when it doesn\t work it is more of a management issue on the side issuing the data.

To sum this up, I would keep any data enriching out of the ESB and use the ESB trasformations only for format transformations and version transformations (**). One more benefit to keeping the ESB down to routing and format transformations would be that its role is well-contained and it doesn’t turn into a wild-card (does this particular problem gets handled by the ESB, how, why, when, etc…) making your IT assets more manageable. This results in a neater division of concerns in your overall enterprise architecture.

* It is advised that the format of these messages is not using APIs but rather text-based protocols such as XML, pipe-delimited, etc… API couplings is one of the most horrendous things that I have seen, synch-ing libraries on multiple endpoints at the same time is an incredibly time-consuming activity.

** Version-based transformation are transformations of the same message between 2 different versions of it: let’s say that one message producer decides to publish a new version of the messages that it has been produced so far, the ESB should be made aware of this and deliver the new version to the consumers that are registered to receive it as well as create a new message with the old version out of the original message and send this new message (formatted according to the old version) to the clients which are still registered to receive the old version. Part of this versioning effort is the transformation from the current version to an older version and this transformation would be best done by the message producer which will load a transformer into the ESB for this transformation. The ESB will then be re-configured to apply this transformation to the original message with the new version and the resulting message to be routed on its old route. This way the versioning effort will be pushed inside the ESB and messages will be delivered transparently to the intended receipients and recipients will not have to be upgraded when their counterparties upgrade their message format effectively de-coupling their release cycles.