From my experience I see that replication of data usually helps deal with performance problems, but that sometimes this replication involves replication of functionality, sometimes you end-up replicating some data on a different system (let’s call this the client system) and then you find yourself needing to code on the client system some piece of business logic which resides on the system from where data comes originally from (let’s call this system the master system).

More often than not this is a pretty scenario, because every modification to the business logic done on the master system has to be re-coded on the client system. Sometimes you find that you need to coordinate the releases of these 2 separate systems in order make the whole picture functionally consistent. One way to avoid re-coding the business logic on the client system is to expose the business logic on the master system thru some remoting mechanism (EJBs, Hessian, etc…) and then have the client system pass the data that it replicated locally to the master system in a synchronous call. While this solves the fact having the same business logic reside in 2 places it makes the client system dependent on the master system (*).

One solution to the above conundrum that is to replicate the data, along with the results of carrying out that piece of business logic on that data, to the client system. With this arrangement the client system does not need to have the master system up and running and it also does not need to re-code the same business logic inside. If you manage to use this (this approach doesn’t work in all cases) you will find yourself exporting business logic asynchronously by exporting both data and the results of carrying out that business logic on the data.

If I don’t write another post till January I wish you a Happy New Year!!!!!

* There is a physical dependency (the client systems needs the master system up and running for it to service calls) and possibly a library depedency which may couple the release cycles of these 2 systems (if the client system communicates with the master system thru objects rather than thry wire protocols you will need to synchronize the release cycles of these 2 systems in order to make sure that the client system doesn’t run in PROD with obsolete libraries).

Typically knowledge is considered to be an asset, some going as far as saying that knowledge is power. Well, when you design interactions between systems is better to avoid systems know to much about each other. Having deep knowledge about a system with which you are integrating is both a moral hazard (it gives you the opportunity to hack your way out of problems) and creates high transaction/interaction costs (typically the costs of transferring this deep knowledge to someone else).
The same reasoning goes for interaction between modules in a single system, please check out this article

P.S. I was shocked to see the date on that article: 1971. The fact that we are still struggling with these problems shows a pretty big problem with computer science. Most people equate computer science with exotic algorithms, it would be about time to add to this equation software modelling. It is pathetic when useful concepts for writing software developed 30 years ago are still unknown to the typical developer.
Software development has a pretty big educational problem (most developers educate themselves on their own and learn on the job), it would not be a bad idea to address it.

Relational databases are going thru a rough patch right now, some pundits going as far as writing them off. Their main problem is the fact that they are constrained to run within the same physical box (*) and scaling out is pretty hard. Once the datasets reach a certain size you will probably need to look beyond the typical relational database at other ways to store your data.

One type of alternative datastore that is getting a lot of attention is the distributed key-value store which maps values to keys and then assign keys onto multiple storage nods according to a hash function (**). In such a setting you get an object thru a key, you work with it and then you save it back in the datastore. The fact that this configuration scales out easily (***) makes this datastore very appealing, and if you are working with large datasets you will probably have no choice but to use something like it.

The transition from relational databases to key-value stores will probably include taking relations out of your application, re-modelling your application in order to group data together and streaming data to a reporting database. Relational databases gave the user both data storage as well as relations between entities which came as both a blessing and a curse (while relations made reporting easier they also gave un-checked access to data which allowed all sorts of corners to be cut). Well, key-value stores will take relations out: data access is more local, you typically have access only to what is immediate to a particular entity and you cannot perform queries spanning your entire data model. This type of data access could turn out to be actually a good thing, because these very close relationships will force an application to have a cleaner design since you will not be able to rely on monster SQL statements for compensating for design deficiencies.

In an application which needs to handle massive amounts of data relations between data will be delegated to activities which require them (reporting is a pretty good example) and data will be streamed from the key-value stores to the relational databases where these activities take place.
All in all, a pretty good division of tasks: applications with a (hopefully) cleaner model relying on key-value stores streaming data to relational databases for reporting.

All the above doesn’t mean that relational databases will disappear, the vast majority of applications do not require to process the massive amounts of data which render your typical database unpractical. Relational databases will be around for quite a while.

* Obviously, there are database clusters (such as Oracle RACs), but they are pretty costly.

** Examples of key-value stores are Google’s Big Table, Amazon’s Dynamo and IBM’s eXtreme scale.

*** Actually, there are constraints attached to it. In order to get the best performance you need to model your application so that data access occurs within the same node in order to avoid a distributed transaction spanning across multiple nodes. Please read this article by Billy Newport from IBM for a better understanding.

In a previous post I was saying that some technological issues are simply proxies for organizational issues and that whether one techonology will thrive or fail within an organization is not determined exclusively on the technology itself but it could be influenced by the organization itself. I ended the post saying:
For example, you may find out that DSLs will not work in particular setting not because DSLs are bad, but because the organization within which these DSLs are implemented and used simply doesn’t warrant their success.

I will get into an example where using DSLs is probably not a good fit because of organizational issues rather than inner strengths and weaknesses. Let’s say that you have some distributed teams working on the same application and that these distributed teams are led by a team of architects which need to create an environment in which these distributed teams work together. I find it pretty hard to see how this environment could be constructed using DSLs, particularly iteratively-developed DSLs, because changes in the DSL would have to be propagated across multiple teams in dispersed locations which would need to be re-trained for every iteration. Compare this approach with creating an API that would be released every so often and for which re-training costs are very small.

I have the impression that the main cost of DSLs is the distance between the place where the DSL is produced and the place where the DSL is consumed and that DSLs tend to be either specific to the location where they are produced or widely known and used within a community (in the second case the distance between the DSL producer and the DSL consumer is compensated by “mind-share”). I would say that the main difference between a DSL and an API is that DSLs are more tightly bound to the domain (by definition) and that part of the costs of using a DSL is the cost of transferring the knowledge about that domain between various stake-holders. This would mean that domain experts can adapt to a DSL targetting their domain faster than a typical developer that is new to the domain and that domain experts would be more productive. The difference in productivity between various members of the work-force means that the work-force composition should partly drive the decision to choose a DSL or APIs on a particular project. Regarding the distance between DSL production and DSL consumption this paper discussing how distance affects knowledge transfers is probably a good read.

Other issues wih DSLs such as developer lock-in (a developer working with a DSL can become out of synch with the rest of the IT landscape), the lack of language design skills on the IT job market, the problem of capturing domain knowledge correctly in large or rapidly changing domains are not explored in this post.

I end this post hurriedly…

Later Edit: I was watching this interview with Charles Simonyi and I would say I remained with these impressions:
If the DSL comes from an already existing language, such as domain experts notations, then there are chances that the DSL will be a good language (this addresses the issue of lack of language design skills on the IT job market). The problem of creating a new language is done away with by re-using a currently used language (the domain experts notations) and creating a new way to implement this existing language.
Regarding domains which are changing rapidly Charles Simonyi says that by separating the notation from the actual domain schema it becomes pretty easy to change the notation (the DSL) without affecting the underlying domain schema. Creating a DSL becomes an iterative process.
End-user programming is to remain a pipe-dream because of the difference in skills between what is needed for programming and the difference in skills needed for working within the domain. DSLs seem to engage the domain expert more into the process of software creation rather than turn domain experts into programmers. Domain experts contributions to software creation thru DSLs may be simply to communicate the domain better to a programmer. Language work benches would create a new division of tasks according to skills.

LLE: I watched this presentation by Intentional Software on domain work-benches and I remained with these impressions:
In an environment where domain workbenches are used it looks like the programmer will be involved in creating projections to an operational environment. The developer will focus on the systems while the  domain experts will focus on the business logic. This division of tasks addresses the issues arising from developers’ lack of enthusiasm about the domain.
It looks like domain work-benches should be used in large and rapidly changing domains where the transfer of knowledge from domain experts to developers is very costly due to the size of the domain and the speed at which it moves (this answers one of my questions above).

I think that the argument that I was making above that DSLs will be location-specific still applies even with a domain workbench in the picture because knowledge transfers with high costs will not appear in the creation of the DSL itself (the DSL being just one notation for an underlying domain schema, one notation among many, it can be created and used locally) but rather in the creation of the domain schema itself (which may end up being a collaborative effort carried out among people in various locations).

As I was saying above I have the impression that in the near future domain workbenches and DSLs will be used where the costs of knowledge transfers from domain experts to developers is large. If the costs of creating languages start to drop and these domain work-benches become ubiquitous then it is possible that a complete separation between domain logic (stored in domain code and expressed thru various notations) and its implementation in a system (which appears to be essentially a projection of the domain code into an operational environment) will appear. It would be interesting to work with Intentional Software’s domain workbench too.

We will have to wait and see…

L3E: This is another presentation on the Domain Workbench.

Code reuse is one software development goal consistently sought for. Its main benefits are that the code which gets reused becomes an investment on which you can build later. However, code reuse has its limits when the codebase is shared by a large number of participants, limits due to communication costs and to diverging interests.

Code reuse granularity goes against the number of the participants because implementing code reuse means deciding on what is common (and prone to be re-used) and what is specific as well as on the mechanisms used for implementing the relationships between what is common and what is specific (via sub-classing, various design patterns, etc…). This process of decision starts to blur once the number of participants increases and the corresponding communication costs start to rise. Too many people arguing about what is common to all of them will have quite a few problems figuring out what they are agreeing on.

The interests of pool of participants also are also more prone to diverge as their number increases, further reducing what is common to these participants. The end result is that once the number of participants increases both the commonalities between them shrink (as their specific interests diverge) and finding and implementing these commonalities becomes more expensive (because of the higher communication costs which occur between a larger number of participants). In these situations the code which gets reused starts to boil down to the main architecture classes within which the participants to the projects are coding their specific classes.

Keeping the number of participants low (agile techniques suggest keeping the number of participants below 7) is a way to deal with a codebase and a way to encourage code reuse efficiently.
However, sometimes you may end-up with a high number of participants. In this case code reuse should be brought down to re-using the main architecture classes in order to avoid confusion born out of mis-communication.
Another technique to deal with a large number of participants is to partition the code-base in order to minimize the number of people which are sharing code, the less people are sharing code the less the communication costs and the more things they will have in common.
Big, unpartitioned codebases come with big problems from what I have seen so far.

I have avoided reading and writing about SOA lately because SOA has attained buzzword status and the high number of articles written about it makes it pretty difficult to follow. I also see so much repetition of the same terms and concepts that pretty much every article I have read in the last 4 months feels stale once I go over the first 2 lines.

Against this background I have read 2 interesting articles on SOA: one article by Martin Fowler about using open-source techniques in order to implement collaboration between a consumer service and a supplier service and one article about abandoning the mindset which focuses on IT and rather on focusing on the relationships between various services thru contracts and policies (contracts and policies are organizational issues and not technology issues).

I found Martin Fowler’s article refreshing, even though I have to say that I disagree with it. From what I understand from the article. I think that having developers commit code to a service that you need to connect to has high interaction costs (basically the developers need to understand the partner service, need to be coordinated with the rest of developers, etc…), interaction costs which drive down the number of services with which you can partner effectively. Martin Fowler takes the issue of partnering with a service even further by having developers from the service consumer become full rights committers to the codebase of the partner service. When your developers are committers to a different service you have to wonder on what service are these developers working. The service consumers and the service providers are becoming very tightly coupled, not thru the interfaces they use for communicating (REST, WS-*, wire protocols, etc…) but thru the developers that these 2 services are sharing. I would say that Martin Fowler open-source like approach to service interactions does not scale to a larger number of services. I could go a bit into the differences between open-source projects and your typical enterprise application (I think that the most important one is that on an open source project features are chosen by boiling down competing features to the lowest common denominator chosen by the OS project participants while your typical enterprise application chooses its features in order to boost its revenues), differences which will pretty much tell why OS-like projects are very rare in the enterprise computing world but I don’t really have the time.

I have to say that when comparing the above OS approach to SOA to the typical vendor approach to  SOA you see big differences: a typical vendor assumes that services are out there in a registry and that a business analyst can create new processes and new value by drag-and-dropping various services into an IDE storing references to firm-wide services. This vision does not have much in common with the real world for two important reasons: 1) there are interactions costs associated with consuming each service, interaction costs which are not expressed in a service IDE and 2) in order to manipulate services at a firm-wide scope you need to master knowledge at the same scope. Possessing knowledge at this scope is nearly impossible, I personally have seen rather the reverse: business analysts getting so specialized that their scope becomes more and more focused. As far as I see it in order to achieve the SOA-nirvana championed by your typical IT vendor you need significant disruptions in your work-force.
And this brings me to the MWD article which echoes some thoughts that I wrote a while back when I was saying that interactions between IT services will be driven by the interations between the BUs implementing these services. SOA is not about empowering some business analyst to draw some funny diagrams on a board and re-engineer information flows at the firm level, but rather about how you create relationships and interactions, how you create incentives for adhering to rule and penalties for straying away from them, how you delegate managing an interaction to the parties involved in it, how you identify and consolidate common tasks into shared services and how you allocate tasks which are particular to a small set of parties, how you deal with who loses in these shifts, etc…

As you can see, all these concerns don’t have much to do with IT, being first and foremost management concerns. The only thing that IT can do is help these management issue get resolved, but not resolve them. IT brings some capabilities which can be used: service registries (which consolidate the view of and access to IT assets), ESBs (which allow for controlling data flows), policy standards (which allow the automation of defining and enforcing policies), etc… These capabilities should not be mistaken for their usage or their ends.
Ultimately, all the IT issues on SOA will be pushed to the back and management will take over. Ideally this field will become so specialized and its audience so narrow that we will get rid of another tiresome buzzword…

Later Edit: As you can guess this post has been written in about half a dozen different sessions, each sessions adding a few more lines to what was written before. I don’t have much time to write anymore (and I guess it shows), but I felt compelled to comment on those 2 articles because I thought they stand out from the rest of articles written on SOA.

As I was saying in my previous post a lot of technological issues (such as the debate of REST vs. WS-*) are actually organizational issues and that at the bottom of them you will find that the process of communication has been broken down into tasks (namely the task of  the payload-to-processor mapping) and those tasks have been allocated (they have been externalized or internalized) according to the transaction costs of each particular environment.

I would follow this by saying that a lot of time will be saved if the interactions between various parties will be broken down into tasks and that the tasks will be allocated according to the nature of the organization(s) within which these interactions will be carried out. This applies not only message passing but to any other environment in which various entities engage in collaboration. 
For example, you may find out that DSLs will not work in particular setting not because DSLs are bad, but because the organization within which these DSLs are implemented and used simply doesn’t warrant their success.

One of the best presentations that I have seen lately is Mark Little talking about REST and WS at QCon London 2008, or rather about the differences between the uniform interfaces and the specific interface. Mark Little gets right at the bottom of the differences between REST and WS, and these differences are not all differences between architectural styles, but rather differences in organizations.

One of the best interpretations of the differences in REST vs WS that I have seen is the one that Mark makes in this presentation and that I will try to reproduce below is the division of tasks of mapping a message sent to an entity (could be a service, could be an URL) and the component/processor within that entity that will process it: when you send a message to a specific interface you will be specifying the method and the component that will process that message, so the message-to-processor mapping is done by the client. Whereas when you send a message to a specific interface you need to attach some meta-data to that message which will be used by the entity implementing the specific interface for mapping that message to its appropriate processor, therefore the message-to-processor mapping is done by the entity.

The message-to-processor mapping is externalized in the case of specific interfaces while the uniform interface internalizes it. These different mappings come with different transaction costs: in the case of the external mapping the costs are synchronizing the client with changes in the specific interface while in the case of the internal mapping the costs are adapting the uniform interface’s mapper to changes in the meta-data (a new type of messages should be mapped to a new processor).
Now, if we apply economics to the task of message-to-processor mapping we would start to get some interesting results. According to Coase’s theorem you can externalize an activity if the transaction costs for that carrying that activity are low enough and if the transaction costs are high then that activity is better internalized. It would follow that when the transaction costs of this mapping are low you can externalize it via a specific interface, otherwise you would be better off to internalize it via an uniform interface. 
The transaction costs of the specific interface are the synchronization of distributed clients and this could be resolved either thru versioning, or by not modifying the specific interface or by standards. If you can keep these costs down it follows that you can delegate the message-to-processor mapping efficiently to the client else you are better off with a uniform interface. Keeping these costs down also implies that the number of methods that get exposed is kept down in order to avoid the number of changes (more methods, more opportunities for changing them). Generally speaking interaction via specific interfaces is best limited to a few coarse-grained components, between a few participants and with components which do not change very often.
The above synchronization costs run counter to scale in number of clients: as the number of clients starts to grow you are starting to have problems synchronizing all of them and eventually the transaction costs of using the specific interface will become so high, it will not make sense to use it. One corollary of this is that the uniform interface is far better at the scale of the internet and this is probably the main reason why it is used at this scale.
The above will provide a pretty good guide to an architect trying to figure out whether to use a specific interface or an uniform one because their use is not driven by technology, but rather by the organization(s) within which they are employed.

Another interesting point in his presentation was the effort for creating a REST-ful standard for carrying out distributed transactions and the need for agreement between multiple parties. As I was saying in a previous post about REST and WS the differences between these 2 camps are more of organizational nature than anything else. Mark touched upon the fact that even if the need for carrying out distributed transactions via REST exists there is still no standard for doing so, that the REST community seems pretty much happy with implementing low-level infrastructure and not with higher-level, and higher-value, components. I find this a pity and I have the impression that this need will be fulfilled only it raises to a level at which major corporations will find profitable for getting involved. The REST community appears simply way too fragmented to be able to carry out this effort on its own…

Anyway, go ahead and watch the presentation, it is the most thoughtful presentation on distributed computing that I have seen in a while.

ESBs is an essential enterprise computing capability which allows for mediating connections between various parties. Typical EBS usage consists of message producers sending messages to an ESB which transforms them and then forwards them to message consumers.

It is at the transformation stage that the thorniest issues arise with ESBs. As outlined in this presentation from Thoughworks there is a tendency to push business logic inside the ESB resulting in a very tight coupling between applications and the ESB (part of the business logic will be stored in applications, part of it in the ESB and the business logic in the ESB will need to be kept in sync with the business logic in the applications). Not a nice scenario and unfortunately the solution provided in the presentation (apparently you only need to use Guerilla SOA) doesn’t say much. The business logic which gets shoved into the ESB in the above example are the relationships between the various parties which are sending each other messages and the mess that comes out of it as these relationships change over time.

One example of ESB usage is a customer-facing web application which takes purchase orders from customers and sends messages to an inventory and shipping system, to the accounts receivables system and to the customer loyalty system. There are a few ways that this messages can be sent: 1) the web-application sends only one message to the ESB which then creates 3 more messages, one for the inventory system, one for the accounting system and one for the customer loyalty system (in which case the relationships between the web-app and the 3 systems are managed within the ESB); 2) the web-application creates 3 messages which are all sent independently to the 3 systems (in which case the relationship between the web-application and these 3 systems is managed by the web-app typically thru dedicated senders which are transforming the message according to the other party’s specs) or 3) the web-app publishes the message on the ESB which forwards it as-is to the receiving systems which are receiving the message as it has been created by the web-app and transform it according to their rules (in which case the receiving systems are managing the relationships with the web-application typically thru dedicated receiving points which are transforming the incoming messages).
So far I have only seen 2 or 3 used and it was used successfully. I have not seen 1 being used successfully, but rather I have seen ESBs simply delegated to the role of a dumb-pipe while the processing is done on the end-points. I think that the main reason for which the ESB cannot grow beyond the role of a dumb-pipe is that the transformations typically involve data, imagine that in the above scenario the IDs of the products on the web-application are different from the ones in the inventory system (a very common case BTW). In order to service this transformation you would need an up-to-date mapping between all relationships that it is managing. Sooner or later that ESB will turn into a giant vacuum cleaner that will need to suck every byte of data available in order to service the relationships stored in it.
Lesson #1: Managing relationships typically involves managing data.

Let’s go more into the gory details and think about how we would deal with stale data: let’s say that the datastore holding the mappings of product IDs from the webapp to the inventory systems is missing a product ID (another very common case). In the case where the relationships are maintained at the connection end-points (message producers or consumers) the operation team managing those end-points would have access to the exception (missing product ID mapping in this case) as well as to its solutions (create this mapping in your mapping store). How will the ESB deal with stale data? Well, the ESB will have to find the entity which can resolve this exception, package the exception and send it to that entity. Sooner or later the ESB gets becomes a spaghetti bowl.
Lesson #2: ESB has bad exception management capabilities. 

Pushing the transformation towards the end-points effectively couples the end-points because these endpoints must share the format (*) of the messages they exchange and, just as importantly, they need to share data. Sharing data is usually done by one party having a procedure to export its data and the other parties using this procedure for creating local replicas of that data and synch-ing them according to a calendar. In addition to that the parties replicating the data have procedure for overriding the data they receive and adding to it. As far as I have seen it works pretty well and when it doesn\t work it is more of a management issue on the side issuing the data.

To sum this up, I would keep any data enriching out of the ESB and use the ESB trasformations only for format transformations and version transformations (**). One more benefit to keeping the ESB down to routing and format transformations would be that its role is well-contained and it doesn’t turn into a wild-card (does this particular problem gets handled by the ESB, how, why, when, etc…) making your IT assets more manageable. This results in a neater division of concerns in your overall enterprise architecture.

* It is advised that the format of these messages is not using APIs but rather text-based protocols such as XML, pipe-delimited, etc… API couplings is one of the most horrendous things that I have seen, synch-ing libraries on multiple endpoints at the same time is an incredibly time-consuming activity.

** Version-based transformation are transformations of the same message between 2 different versions of it: let’s say that one message producer decides to publish a new version of the messages that it has been produced so far, the ESB should be made aware of this and deliver the new version to the consumers that are registered to receive it as well as create a new message with the old version out of the original message and send this new message (formatted according to the old version) to the clients which are still registered to receive the old version. Part of this versioning effort is the transformation from the current version to an older version and this transformation would be best done by the message producer which will load a transformer into the ESB for this transformation. The ESB will then be re-configured to apply this transformation to the original message with the new version and the resulting message to be routed on its old route. This way the versioning effort will be pushed inside the ESB and messages will be delivered transparently to the intended receipients and recipients will not have to be upgraded when their counterparties upgrade their message format effectively de-coupling their release cycles.

Typically OOP inheritance is used for sharing various common methods and a typical example would be something like this:
You have interface Vehicle which has sub-classes Bicycle and MotorVehicle. MotorVehicle in turn has the sub-classes Car and Truck, each with its specifics. Car, Truck and Bicycle are all sub-classes of Vehicle and as such are all inheriting the same common behavior, such as some object which specifies the speed limits within which each Vehicle should operate (for example a Car should be able to handle higher speeds than a Truck or a Bicycle).
Fair enough, and this way to use OOP inheritance has its reasons for use.

However, one other reason for using inheritance that I see is for creating substitutes in order to partition code bases. In this case you are using sub-classes not because you need some shared behavior to be encapsulated in a super-class, but because you have distributed teams working on the same codebase and you need to minimize collisions between them in the codebase (a collision would be 2 or more persons working on the same file and is usually resolved by a merge in CVS). Solving conflicts thru CVS merge is a pretty error-prone process and doesn`t scale out well, so at one point it is necessary to partition development so that different teams can work on the same project without elbowing each other.
Sub-classing in this case would solve this problem by letting a developer substitute its team’s type for another team’s type.

One example is using helper classes which service a POJO and which are used by more than one team:
Let’s say that we have a TaxProcessor class that uses a TaxCalculator for calculating the taxes to be applied to a trade. Let’s say that you have 2 teams, one working on taxing bond trades and one working on taxing equity trades and that they are all using the same TaxCalculator class on the TaxProcessor. TaxCalculator has the method taxTrade which delegates to taxBondTrade and taxEquityTrade. As time goes on a lot of fixes are put into taxTrade in order to deal with various corner-cases to the point where 1) TaxCalculator becomes hard to maintain and 2) more than one developer is working on the same method at the same time. A solution would be to sub-class TaxCalculator into BondTaxCalculator and EquityTaxCalculator, have each implement the method taxTrade according to its specifics and then have TaxProcessor use the appropriate class according to the trade type.

As you can see from above good software models imply good separation of concerns. Good separation of concerns translates in good separation between development teams working in parallel on the same project. Good separation between development teams results in low transaction costs or interaction costs between teams and into a more efficient way to delegate work and in a development environment which scales better.