As I was saying in my previous post a lot of technological issues (such as the debate of REST vs. WS-*) are actually organizational issues and that at the bottom of them you will find that the process of communication has been broken down into tasks (namely the task of  the payload-to-processor mapping) and those tasks have been allocated (they have been externalized or internalized) according to the transaction costs of each particular environment.

I would follow this by saying that a lot of time will be saved if the interactions between various parties will be broken down into tasks and that the tasks will be allocated according to the nature of the organization(s) within which these interactions will be carried out. This applies not only message passing but to any other environment in which various entities engage in collaboration. 
For example, you may find out that DSLs will not work in particular setting not because DSLs are bad, but because the organization within which these DSLs are implemented and used simply doesn’t warrant their success.

One of the best presentations that I have seen lately is Mark Little talking about REST and WS at QCon London 2008, or rather about the differences between the uniform interfaces and the specific interface. Mark Little gets right at the bottom of the differences between REST and WS, and these differences are not all differences between architectural styles, but rather differences in organizations.

One of the best interpretations of the differences in REST vs WS that I have seen is the one that Mark makes in this presentation and that I will try to reproduce below is the division of tasks of mapping a message sent to an entity (could be a service, could be an URL) and the component/processor within that entity that will process it: when you send a message to a specific interface you will be specifying the method and the component that will process that message, so the message-to-processor mapping is done by the client. Whereas when you send a message to a specific interface you need to attach some meta-data to that message which will be used by the entity implementing the specific interface for mapping that message to its appropriate processor, therefore the message-to-processor mapping is done by the entity.

The message-to-processor mapping is externalized in the case of specific interfaces while the uniform interface internalizes it. These different mappings come with different transaction costs: in the case of the external mapping the costs are synchronizing the client with changes in the specific interface while in the case of the internal mapping the costs are adapting the uniform interface’s mapper to changes in the meta-data (a new type of messages should be mapped to a new processor).
Now, if we apply economics to the task of message-to-processor mapping we would start to get some interesting results. According to Coase’s theorem you can externalize an activity if the transaction costs for that carrying that activity are low enough and if the transaction costs are high then that activity is better internalized. It would follow that when the transaction costs of this mapping are low you can externalize it via a specific interface, otherwise you would be better off to internalize it via an uniform interface. 
The transaction costs of the specific interface are the synchronization of distributed clients and this could be resolved either thru versioning, or by not modifying the specific interface or by standards. If you can keep these costs down it follows that you can delegate the message-to-processor mapping efficiently to the client else you are better off with a uniform interface. Keeping these costs down also implies that the number of methods that get exposed is kept down in order to avoid the number of changes (more methods, more opportunities for changing them). Generally speaking interaction via specific interfaces is best limited to a few coarse-grained components, between a few participants and with components which do not change very often.
The above synchronization costs run counter to scale in number of clients: as the number of clients starts to grow you are starting to have problems synchronizing all of them and eventually the transaction costs of using the specific interface will become so high, it will not make sense to use it. One corollary of this is that the uniform interface is far better at the scale of the internet and this is probably the main reason why it is used at this scale.
The above will provide a pretty good guide to an architect trying to figure out whether to use a specific interface or an uniform one because their use is not driven by technology, but rather by the organization(s) within which they are employed.

Another interesting point in his presentation was the effort for creating a REST-ful standard for carrying out distributed transactions and the need for agreement between multiple parties. As I was saying in a previous post about REST and WS the differences between these 2 camps are more of organizational nature than anything else. Mark touched upon the fact that even if the need for carrying out distributed transactions via REST exists there is still no standard for doing so, that the REST community seems pretty much happy with implementing low-level infrastructure and not with higher-level, and higher-value, components. I find this a pity and I have the impression that this need will be fulfilled only it raises to a level at which major corporations will find profitable for getting involved. The REST community appears simply way too fragmented to be able to carry out this effort on its own…

Anyway, go ahead and watch the presentation, it is the most thoughtful presentation on distributed computing that I have seen in a while.

I am watching with a great deal of interest developments in the grid and large scale computing environment because I always found distributed computing interesting.

One very interesting thing that I come across pretty often is the fact that in large scale computing data tends to get de-normalized, basically it grows so large that you cannot fit the entities and the relationships between entities in the same database (this infoQ article is pretty good).
Let me give an example: suppose that you have table of users and a table or messages between users having these columns:
users: userID int, firstName String, lastName String, email String
user_messages: fromUserID int, toUserID int, messageTest String.
In the initial design a message from user A to user B would have only one record in user_messages and each user will be able to get the messages it has sent and the messages it has received from this table.
Now let’s say that the number of users sky-rockets to the point where you need to partition your database horizontally into a series of identical replicas. The problem that you will face now is where to store the data that is shared between these replicas, namely the table user_messages. You cannot store it either in the replica hosting user A’s data, neither in the replica hosting user B’s data and neither in its own instance because it will grow too large and because you will need to carry out a join over 2 physically remote databases. The solution is to drop user_messages for received_user_messages (from_user_name, from_user_email, message_text) and sent_user_messages (to_user_name, to_user_email, message_text) which will store the messages that a user has sent and the ones that it had received. Each database replica will have its own instance of these 2 tables each mapped to the user table. As you can see we are not storing the userID in the message tables, but rather the user information which was previously retrieved via joins.
Sending a message from user A to user B in this environment typically involves updating 2 tables in 2 separate database instances which usually is treated as a distributed transaction. However, you cannot carry out this distributed transaction because it is very costly so sometimes you resort to updating one table and sending a message to update the second table asynchronously (the overhead for delivering a JMS message is way less than the overhead for locking rows in the second database). In the case of updating the second table asynchronously you are still enforcing the relationships between the user table and the user message tables, but at a later point in time.
De-normalized data comes with synchronization costs: when you are updating the user information on the users table you will need to propagate the changes to received_user_messages and sent_user_messages so that the data stored in these tables will be up-to-date. This could be done via asynchronous processing as well, sending a message about data being changed on an ESB and having the concerned parties listening for and processing it. Synchronization costs should be watched very carefully because they could spawn a very high number of messages. Ideally we would synchronize data which is updated rarely (such as user information) in order to keep the number of synchronization messages down. Carrying out synchronization procedures in batches could be a way to deal with a large number of synchronization messages with the side effect that it may increase the latency with which the synchronization takes place.

Large scale computing typically seems to resolve around asynchronous processing (because in transactions message passing is cheaper than database access) and de-normalized data across with the overhead implied by it. The relationships previously enforced by normalizing data are usually enforced by passing JMS messages which carry out the data changes asynchronously. This design is driven primarily by the large volume of data which cannot be serviced by only one database and by the costs of carrying out a distributed transaction spanning 2 databases.
This is one design that I see emerging for large scale computing: de-normalized data along with asynchronous processes which are enforcing the relationships between entities via message passing.

It remains to be seen if this type of grid architecture will prevail in the future. Working against it is the emergence of multi-core processors which would allow for scaling up cheaply as envisaged by Brian Goetz in this article. If chips with hundreds of multi-cores and with large amounts of memory become reality scaling out could end-up costing more than scaling up and all the above could become history pretty fast. Scaling out will continue to make sense in some environments with humongous amounts of data (think Amazon, Google, eBay, etc…) but for the current fastest growing segment in grid applications, typically applications processing moderately large amounts of data, it may make sense to scale up once chips with a large number of multi-cores become a reality.

To watch…

P.S. These shifts in data processing (moving to a grid-type of architecture in order to accommodate the increase in data and then back to a single-box architecture because of the appearance of chip with a large number of cores) makes a pretty good case for abstraction in order to lower the costs of transition from one architecture to another. Ideally the architecture and the environment in which an application runs should not affect the business logic of that application. One way to insulate your application’s business logic from these infrastructure issues is by abstraction.

I was watching this infoQ presentation by Alexandru Popescu and Cedric Beust on testing when I realized that a big market for testing is seriously neglected.

Regarding my use of testing I would say that JUnit pretty much fills my bill and I don’t see a need to move to TestNG. I was watching the presentation not so much as to know more about TestNG, but to get some exposure to the market for testing products (*). So the features which were presented and which were said to be in a high demand from the users were the ability to define test groups, to test data connectivity and to define dependencies between tests (**). Pretty fair, I would take this functionality to be a logical progression from the ability to run a bunch of tests at random as JUnit lets you do it, you are basically starting to look at the tests you run from a higher level and you start looking for ways to aggregate these tests into higher-level constructs, such as test groups, which can then be manipulated according to various needs.
So far I would say that Test NG is looking at ways to manage the complexity coming out of a high number of tests. Test NG is for test-heavy shops, where testing is considered a concern on par or close to development. Not a bad thing and I pretty sure that TestNG covers some functionality which is in high demand by Java developers.

There is, however, one need for testing that so far it is largely unsatisfied and sorely missed: the need to test workflows or series of events. Let’s say that you have an application that is a series of MDBs, each accepting messages, transforming them and then outputting them to the next MDBs. You would want to be able to test this application end-to-end.
Let’s say that you have an application that receives market trades, needs to process them and then transform them in order to send them downstream to tax systems, settlement systems, etc… You would want to put thru a trade, set-up in a certain way and then trace its execution thru these flows and determine if, in what stage and in what shape has this trade reached the Application-to-Tax-System gateway.
When I think about work-flow testing I usually think about defining a message, inputting this message into a work-flow engine and then defining interceptors to see how the original message has been manipulated at various stages. You may need to test both the work-flow itself (to see if the message has been going thru the work-flow that it needed to go thru), the end-result (to see where the message has been forwarded to and in what shape) as well as how it behaves at various stages in this workflow (if needed). I see work-flow testing primarily concerned with interception of messages (and this would probably be a great use of AOP) and with the possibility to correlate messages passing thru this work-flow with the original message.

OS testing tools so far are limited to synchronous testing. I wonder at what point will the need for work-flow testing become so pressing and the demand for it so great so that one OS shop will start doing something about implementing a testing framework for testing workflows. Then we could use Event-Driven-Testing for testing an application written in a Event-Driven-Architecture manner…

* I know that trying to form an opinion on a market such as the market for OS testing tools from a vendor presentation is a pretty risky business, what you get from vendor presentations is usually distorted because of bias and time constraints but I will assume that this presentation gave a fair image of what users want from testing tools.

** From what I know TestNG is a lot more than some annotations that give you the possibility to form test-groups and test dependencies. However, I would say that these issues are probably considered more important and more aligned with the market for OS testing tools since they were the ones which were included in this presentation.

P.S. If I were to choose one enterprise concern which is not addressed by the Spring stack I would choose work-flow. Spring has modules for integration, batch, transaction management, security management, connectivity to various end-points, etc…, it is missing this capability. From what I see work-flows are used pretty heavily in the enterprise space and for now I think they are mostly implemented either by in-house solutions or by commercial solutions. OS could probably make a contribution in this space as well. And if it does it should probably try to give the developer the ability to test work-flows, or at least make it easy for him/her.

P.P.S. The comments by Alan Keefer on this thread reinforce my beliefs that we need to carry out tests at every level, from high-level to unit test. For a work-flow based system this would mean that we need to test the work-flows themselves and not only the units making them up.

Later edit:  There are actually a few OS workflow tools. OSWorkflow, the ones at the bottom of this article and probably a few others. I should try them at one point.

I was reading this post on infoq about how good object-oriented programming mimicks a lot asynchronous processing. It is a pretty interesting post, even though it is not very organized, the author surfs around a few concepts, finds some interesting relationships between them and then ends the article.

The post looks for similarities between implementing some piece of functionality in an asynchronous manner and implementing the same piece of functionality with methods that do not return values. It pretty much finds these similarities, but there are some important differences which we will explore now.
Let’s take a look at some code: suppose the you have this purchase order component which has the method placeOrder for placing an order:
public PurchaseOrderBean{

public placeOrder(PurchaseOrder po) throws PurchaseOrderException{

try{

// update its status in the DB

changePurchaseOrderStatus(po);

// execute payment

paymentBean.performPayment(po);

// update the inventory

inventoryBean.updateInventory(po);

// integrate with the back office

backOfficeBean.notifyBackOffice(po);

}catch(BackOfficeException ex){

// handle BackOfficeException, possibly rolling-back the current transaction

}catch(InventoryException ex){

// handle InventoryException, possibly rolling-back the current transaction

}catch(PaymentException ex){

// handle PaymentException, possibly rolling-back the current transaction

}

}

}

As you can see in this case the method placeOrder simply tells the inventory bean, the payment bean and the back-office bean to do their work.

Now let’s implement the same method in an asynchronous manner using JMS and QueueSenders:
public PurchaseOrderBean{

public placeOrder(PurchaseOrder po) throws PurchaseOrderException{

try{

// update its status in the DB

changePurchaseOrderStatus(po);

// execute payment

queueSenderToPaymentJMSQueue.send(formatMessage(po));

// update the inventory

queueSenderToInventoryJMSQueue.send(formatMessage(po));

// integrate with the back office

queueSenderToBackOfficeJMSQueue.send(formatMessage(po));

}catch(JMS exception){

// handle JMS Exception. Usually it means rolling-back the transaction and sending this exception up-stream.

}

}

}

As you can see the biggest difference between the 2 implementations is exception handling. In both cases you are telling a component (in the first case a bean, in the second case a different system) to carry out its work but in the first case you get to deal with the exceptions. One big difference between asynchronous processing and synchronous processing that I see is that in the case of asynchronous processing you are delegating the whole implementation of a particular functionality including failures while in synchronous processing you can delegate dealing with failure to the method that orchestrates the interactions between these components.
I consider this difference to be important when designing systems because it essentially splits tasks and responsabilities.
The post was right on the money on the need to try to avoid methods returning values which need to be dealt with in the calling methods, because coding in such a way encourages to write components to which you can delegate a task better. Having to deal with methods that return values means basically that you need to interpret these values outside of this method and this means that not everything was delegated to the method. Sometimes you need to code this way though…

Now let’s dig a bit into the way exceptions are handled in this 2 pieces of code. Let’s suppose that the payment fails for this particular purchase order, the credit card is expired. In the first example all processing would basically stop at this step, we would roll-back the transaction and notify this error upstream.
In the second example it is a bit more complicated, because the payment would fail in the payment system which would then have to notify the back-office, the inventory system that that particular purchase order has failed (probably the most efficient way to do this would be to publish a message about a failed payment on an ESB channel to which the other 2 systems are listening. Connecting the payment system directly to the systems it is related to is a tight-coupling and should be avoided. *).
Exception handling is done differently in the 2 implementations: the first implementation it deals with the exception right at the source while the second implements exception handling by passing messages between the systems in the back-ground. The second implementation is also more prone to tight-coupling between systems.

One cost involved in the above implementations is the cost of exception handling . In synchronous processing exception handling is done upfront and it is usually not expensive since you are handling it at the source while in asynchronous processing exception handling usually ocurs behind the scenes with systems passing each other messages, you should pay attention to it when designing systems. If you could delegate the whole processing, including failure handling to a component it would probably make sense to use asynchronous processing. If you find that some failures imply multiple systems it would probably make sense to use synchronous processing.

Another cost involved in the above example is the cost of locking down resources: in the synchronous example this cost is pretty high: you need to start a transaction, start processing, lock down database rows, etc… and roll back that transaction in the event of failure while in the second example this cost is extremely low: you start a transactions, you send some messages and you commit the transactions.
You essentially have these 2 costs and each implementation has a different cost: the asynchronous implementation has a high complexity cost (passing messages about exceptions behind the scenes) while the synchronous implementation has a high physical cost (the cost of locking down resources pending a transaction commit). These cost structure should be kept in mind when designing systems (**).

I would end this post saying that while most of the people seem to associate asynchronous processing with low-latency method calls, delayed execution and relaxed time constraints it is important to bear in mind that asynchronous processing should also be associated with the way you delegate a particular task to a particular system which is primarily a management issue. What I wanted to stress is that asynchronous messaging forces you to fully delegate processing (including exception handling) to a different component or system. It is this hand-off that I see as just as important as the other things that asynchronous is associated with: faster return from the call because you are simply passing a message rather than waiting for its execution,etc….

* One caveat though. It is a good thing to try to keep the number of the message types that are sent thru an ESB under control, otherwise you will end-up with a Spaghetti Oriented Architecture ;-).

** The present push away from ACID transactions towards systems passing messages between each other would imply that the physical cost is larger than the complexity cost. At the same time ESBs are becoming more and more popular in order to keep the cost of the relationships between various systems low.
Transactions can be viewed as a way to enforce state synchronization between multiple actors: these actors try to carry out a series of actions at the same time and if one of them fails they should all go back to original state. ACID transactions are enforcing state synchronization at the time the actions are carried out, however in some cases you can carry out this state synchronization later.
Please check these links for some unusual transaction mechanisms. Some of them (especially XTP) will very likely become de-facto standards:
Are Cross-Service Transactions A Violation of the Autonomous Tenet of Service Orientation?
Xtreme Transaction Processing under SBA terminology
Large-scale distributed transactions
Constrained Tree Schemas (CTS) and applications (CTA) for extreme OLTP (XTP)

Note: This post was written while listening to Mylène Farmer live at Bercy 2006.

P.S. This post is also quite all over the place: exception handling in async interactions vs. synchronous processing, transactions, etc… I guess Mylène Farmer can be quite distracting ;-).

If you are to read most of what is published today about the Java platform it seems that the future of Java doesn’t look pretty good at the moment as it keeps losing battle after battle with the movement behind dynamic languages such as Ruby and PHP. At the same time improvements to the language which could give it some boost seem to be badly implemented, most people beeing unhappy with the implementation of generics and the proposed closure enhancement looking pretty horrific.

You have to wonder what will happen to Java in the future. Will it disappear and turn into a dinosaur that could not adapt to a changing IT environment or it will manage to survive the problems that it has at the time? One good look at the Java language would have to consider both the language itself and the libraries/frameworks built under it, the contributors to the language, both contributors to the language features and to its libraries/frameworks, etc… as well as the users of the language and its libraries, i.e. the developers that create Java applications.

The different types of contributors to the Java language. I would start by looking at the contributors and I would split them in 2 camps: corporate contributors and non-corporate contributors (*). Looking at the chief contributions these 2 types of contributors make to the Java platform I would say that the corporate contributors are mostly active contributing libraries and frameworks and that the non corporate contributors are contributing Java language variants such as Groovy and Scala (**). The split between contributors seems to mirror the split between the language itself and the libraries/frameworks being built under it, this is an important point.

It is important to look at the motivations of each contributor type for contributing to the Java platform: the corporate entities are mostly contributing libraries and frameworks that target specific problems with a wide audience while the non-corporate entities typically target problems with small audiences, such as languages. The size of the entities involved in a particular task is usually a good indicator of its audience and of the need for coordination (the bigger the number of entities, the bigger the need for coordination).
Corporate entities are more effective in projects where a certain amount of discipline and coordination is required (such as when defining the WS-* specs), while non-corporate entities are more effective in projects which do not require a large amount of coordination. The need for coordination between corporate entities is primarily driven by the number of these entities and the fact that these entities have different, sometimes competing, interests to which a common denominator has to be found.
Opposite to the need of coordination you would find most entities that created the languages currently running on the Java platform: These languages are initiated by individuals and are maintained by a single team that is usually pretty small.

One important thing about the contribution of corporate entities to the Java language in terms of libraries/frameworks is that these corporate entities need these huge investments in libraries/frameworks to be relevant in the future. Backward compatibility is very much needed for them in order to provide the stability required for continuing to make these contributions, this is very important to consider when thinking about truncating Java (see small section below) and when making language changes.

The mushrooming of JVM languages (***). You would expect that once the JRE got modified to allow easier access to different languages on the Java platform that the number of Java variants will grow very large, to the point of becoming unmanageable. This has not happened so far and if I were to identify the reasons for this I would first say that 1) there is an entry barrier for creating a language, you need to design it well and 2) the need to keep relevant investments made when committing to a particular language. The small number of languages used on the Java platform, in sharp contrast to the number of Lisp variants, is due to the fact that when a entity adopts a such a language it will make a commitment to use it in order to keep development costs down (you don’t want your web development shop to work 10 different languages, each needing a guru). The need to keep development costs down, which doesn’t exist in Lisp’s world because Lisp is used by academia rather than by everyday coders, put a ceiling over the number of languages running on the Java platform. I would expect this pressure on the number of languages running on the Java platform to exist in the future, new JVM languages will start being used only if they truly offer gains(****).

Open sourcing Java. Just like the mushrooming of Java languages the mushrooming of Java-like languages which branch out of the main Java language maintained by Sun didn’t happen (looking at the hundreds of Linux variants you would have thought that it should happen). I think that this is due to the exactly same thing that prevented the mushrooming of JVM languages - the commitment that using a language entails.

Truncating Java. The JRE got too big and even with today’s networks it is still a pain to download and install it. Split it up in OSGi bundles and spin them off the runtime (with Java AWT and Swing being the first victims). Keep in the runtime only what is absolutely necessary for the current enterprise libraries to work well (the collections library, threads, etc…).

Competition from Ruby. Ruby appears as a serious contender to Java, supposedly gaining market stare and mind-share. I don’t think that this will last long primarily because Ruby will not have resources for re-creating the current Java libraries. I also think that re-creating Java’s libraries in a different language is a waste of time. My opinion is that it is beneficial to learn Ruby in order to use JRuby and tap into the vast libraries currently available on the Java platform.

My opinion on Java’s present and future. I think that the current split in contribution to the Java platform (corporate contributors generating libraries, frameworks, products, etc… and the non-corporate contributors creating languages) is correct. Let each camp go forward in its own way, the corporate entities will continue to produce the specs, libraries and frameworks that we all use and that we all need to be relevant in the future while the non-corporate entities will work on creating new languages from which to call those libraries. There will not be a mushrooming of languages because the costs associated with using a language will keep the number of languages down. There will be some healthy competition from various languages that attain buzzword-status, but it will not last for long.
This would conclude my post. I admit that its subject (Java’s present and future) is very broad and covering it in one blog post is very hard but I have these opinions that I want to share with the world.

* By non-corporate entities I mean most small cohesive groups that grow around Java such as the groups maintaining Groovy, Scala, etc… By corporate entities I mean both large corporations involved in various specs (such as IBM, BEA, Oracle, etc…) and small corporations such as Spring Source or Red Hat. Also, the contribution of a corporate entity to the Java platform is not restricted to the contribution that entity makes in the open-source space, but it means all the code that entity has created in the Java platform, closed source as well. I know this definition borders somewhat on using the language rather than contributing to it, but I will keep it this way.

** I know that Java languages are also contributed by corporate entities with JRuby being contributed by Sun being one such example (Sun actually brought JRuby under its umbrella, JRuby being started by a non-corporate entity).

*** By JVM languages I mean languages different from Java that run on the JVM such as Grrovy, Scala, JRuby, etc…

**** Interestingly enough, Java platform languages would be a pretty good case of study for word-of-mouth advertising: some guy created Scala, another guy tried it and blogged about it, few more did the same, some other guy requested some features, etc…, before you know Scala is slowly being shoved into the spotlight.

Probably the most interesting thing that I read in quite a while is the paper “The Nature of the Firm” ** by Robert Coase which can be viewed here. Ronald Coase answers a pretty interesting question: why in some cases it makes sense to gather resources together to carry out an economic activity while in other cases it makes sense to delegate that altogether to an outside party.
The answer lays in the extra costs that lay in carrying out a transaction or, as they are known, in transaction costs. Let’s say that you want to have your own house. You have 2 choices: one is to sub-contract parts of building the house and orchestrate the activities of the companies that you have sub-contracted: the one that pours the foundation, the one that puts up the walls, the people that paint your house, etc… or you can delegate the operation to the external entity, the home builder. The costs of each choice are (at first sight) the following:
1) carrying out the orchestration on your own: paying for the company that pours the foundation, the one putting up the walls, the one putting up the roof, etc…
2) delegating this to the home builder and paying for all of the above (the home builder will have to pay the sub-contractors just like you do) + the profit of the home builder who will be doing the orchestration for you.
At first sight it appears that you should be doing all the stuff yourself and come off cheaper than by delegating this to the home builder. But apart from the costs of the subcontractors you also incur the costs of interacting with them: you need to inspect the work of each one because you don`t know how good that sub-contractor is, you need to look around for good subcontractors, you waste a lot of time negotiating with them, etc… No suprise that most people prefer to go with a builder***.

What basically The nature of the Firm says is that if the costs of carrying out the activity on your own are less than the costs of delegating this to a dedicated party you should carry out this activity on your own rather than delegate it. You can see Coase’s theorem at work in the current outsourcing/offshoring trend: the transaction costs for outsourcing some operations are so low to the fact that a corporation can delegate efficiently various tasks to outside partners.

So how would this apply to software development? I would say that transaction costs demarcate pretty well what needs to be grouped together and carried out by one entity (which assumes a specialization of some sort) and what needs to be delegated to a different party. Well, this is what is needed for architecting a project: you need to determine how to group various classes, packages, systems together so that they minimize the interactions between them. The interactions between systems (or classes or packages) can be thought of as transactions: one class calls the other to do some work just like a home-builder calls the carpenter to install the hard-wood floor. By looking at the transaction costs between various components of an application you could determine what transactions (method calls) should be consolidated into component that handles them and what transactions should be allowed to exist on their own.

I would say that one big cost associated with a method call (or call to a package/outside system) is extensive orchestration between various classes (a method that calls 15 other methods from 7 different classes). If you have a pattern of method calls repeating over your code-base you should probably try to take a step back and consolidate these calls into a component that will serve the purpose of that particular orchestration. This way you will create a specialized component to which you delegate this orchestration, rather than replicating it multiple times. Efficient specialization of the components will incur lower transaction costs because it will push down on the number of interactions between components. For this you need domain knowledge. While designing a system you should look for specialization and for ways to achieve this specialization and once this specialization achieved you should look for ways to interact with these specialized components efficiently. If this is done properly you will end up with a system in which the cost of interaction between various components is low.

Another big cost for interacting with a different system is knowledge about the other system and class. The more knowledge you need to have about an external entity the more costly it is for you to interact with that system: you will need to be kept up-to-date with changes in this knowledge, there is a ramp-up cost, you can handle only this much knowledge, etc… It is a good thing to minimize the knowledge required for interacting with an external entity, this way a component can interact more efficiently with other components.
One corollary would be that you need to externalize as little as possible from any component and try to keep most of its sub-components private. One side effect of having too much of a component available is that you run the danger of a component coupling to it unnecessarily.

One cost for interacting with an external system is API coupling: the fact that you need classes exported by that external system**** in order to connect to it. If your system shares classes with an external system you will need a way to synchronize the shared classes with the external system. This is done usually either by having the external system having different versions for its points of interaction or by having a mechanism that publishes classes used for interaction. Either alternative is pretty costly, so you may try to keep away from sharing classes with an external system (though sometimes it may be impossible).
Transaction costs can be applied in project management as well: you should look out complex interactions between teams and try to assign them to a resource best dedicated to handle them (this resource could be a team-member, a well-defined process, etc…). However, in this case you should avoid assigning this interaction to a real person because you will run into scarcity constraints (i.e. you will run out of people) very fast if you keep delegating to real people. This is why these interactions are probably best addressed to a process.

I think that developing a software project while keeping an eye on the transaction costs is a good approach.

Later Edit. Transaction costs are very useful in determining the interaction between large components (systems, services, applications) and they are probably best used for managing this interaction. When you design the interaction of various systems the interaction costs (which are transaction costs) should be kept to a minimum for that interaction to occur without many problems.

* In this post transaction will be used for its economic meaning: i.e. it will stand for an exchange between 2 parties.

** If I am not mistaken Ronald Coase received the Nobel prize in economics for his work on transaction costs and this paper.

*** Please note that there are people that actually build their own houses on their own. The vast majority of them are in the construction business or have connections in the construction business so their transaction costs are a lot lower that for the ordinary person.
On a different note, the orchestration needed for building a home (calling the people that put up the jeep-rock after the 2/4`s have been erected, calling the painter for painting the walls after the walls have been put up, etc…) is a very simple business process without any intrisic value and prone to copying. No wonder that the builder itself makes a pretty small margin while the lion`s share of the profits go to the entity which owns the land…

**** You are generally sharing classes with an external system if you interact with it via RMI or EJB. Interacting with an external system via WS-* or REST is a lighter coupling.

Code contains knowledge about the application and not only the commands to execute that application, this is a thing that is a bit unknown.
Meaningful names for classes/methods, etc… will expose the knowledge buried in the code better to a person working with that code. When you are changing the behavior of a method or a class it is a good policy to also change its name in order to expose the behavior of that class better.
The knowledge of an application should not be encapsulated only in UML diagrams or PowerPoint documents, it should be encapsulated into the code itself because easier to read code translates into a better application.

I have become interested lately into knowledge management. It is a pretty interesting field that applies pretty well to software development.

Later Edit: In the ideal case a person could learn a bit about the domain by going over the APIs, in such an environment you could say that the domain knowledge is pretty well encapsulated in the code. Of course, this would not mean that the domain knowledge should only exist in the code, only that it is a desirable to have the code reflect the domain.

I was listening to this presentation by some people at ThoughtWorks about designing an application when I was struck by this quote which goes roughly like this:
I keep thinking of clever designs to push the technology out of the picture so that changes in the business domain can be implemented more easily.

Very clever quote. The business domain keeps changing a lot more frequently these days, I think due to an increase of the number of stakeholders which increases the variety of requirements, so it is imperative to be able to make those domains changes easily. Being unable to make these domain changes because of technological reasons carries a pretty big cost.
This is why the technology should take a back seat to the domain. Ideally it would be abstracted so that it is decoupled from the domain.

Another quote that I liked was: you are given a domain when you start working on a project, you are not making this domain up and that your implementation should reflect very well that domain. The job of the domain modeler is to find out that domain and then represent it as faithfully as possible. The domain will change, but it not change in a radical fashion because this would imply that the underlying business changes radically and this doesn’t happen that often.
As I was saying before, the domain changes more frequently these days so it is better to have your application very well aligned with the domain so that changes in the domain can be translated into changes in the application very easily. If you cannot add a new feature because some framework doesn’t allow you to then you have a pretty big problem.

Another big theme in this panel was how to defer decisions till the last possible moment, which is a pretty neat things: it pays off to make decisions later when you probably have more information to base your decisions on.

All in all, a pretty interesting presentation.

I was reading this post from Bill Burke’s blog and I found myself shocked by the sheer number of annotations required for making a business object WS-BA compatible (if I understand well the post).
One thing that I don’t like about using annotations is that they tend to stay with the code. I think that these documentation provided by annotations should be outside of the code because in this case you are coupling your business component to an external concern: you are coupling your component to the WS specification.
What you get in Bill’s post is the first stage of annotation creep: you need to externalize your component in order to plug it into an operational environment and you do it thru an annotation. Next operational environment comes with a brand-new set of annotations. And so on…
I keep thinking that the best way to deal with all these annotations would be to sub-class the original interface and apply annotations to it. You will have quite a lot of problems: the classes which are mapped to the original interface will need to be sub-classed as well and their sub-classes mapped to the annotated interface. You will have problems when updating the original interface, you will find that you need to update all its annotated sub-interfaces. And a whole other set of problems.
You feel like you almost need to define inheritance in annotations: an interface would inherit the methods from an interface and the annotations from a different place and you would work with this new interface properly annotated for the operational environment. My opinion is that annotations, like any other type of static, hard-to-change documentation, have serious limitations when it comes to exposing a class to many, varied environments and when you are using it for documenting operational concerns. No wonder so many people prefer XML…

BTW, this post uses Bill Burke’s post only for displaying an example of annotation-creep.

Later edit: If you think about the way annotations are currently used (putting them all over your code base in order to insert a component into various frameworks) they can be thought of as some sort of cross-cutting concerns (they are cross-cutting to a certain extent). I wonder if using something like an aspect would not do a better job of expressing the various concerns that annotations are currently used for. Aspects are orthogonal to the code-base at the same time, a pretty neat thing.
Wouldn’t it make sense to WS-BA-enable your whole application by specifying a point-cut and then applying annotations against this point-cut? It would be a lot easier and it would de-couple your code-base from external concerns. I am not sure if it is doable (I could think of a solution, but it is not pretty).
I think that the way annotations are currently used is not 100% OK and that it poses some maintainability problems in the future. Putting information into annotations is partly a response to XML-creep (as Bill Burke pointed out in the comments) but it is a solution far from perfect. What we are dealing with here is the need to add some information to a component in order to plug it into a framework. Annotations, as well as XML, are not doing a good job because they address this problem (plugging the component into the framework) at the individual level, at the component level. What we really need is a mechanism that would let you plug a whole set of components into a framework easily (I suggested using point-cuts).

Later Edit 2.0: I think that adding querying capabilities at the code-base level (similar to AspectJ`s point-cut language) along with a mechanism to interpret these queries into Java would be the killer feature that Java lovers wait for. It would create a whole new way of developing: for example AOP would be implemented by joining point-cuts to an interception mechanism. A lot of the current mappings (in XML or in annotations) could be solved by defining a point-cut and then a mechanism to interpret this point cut. In the above case you would define a point-cut of your components that you want to WS-BA-enable and a mechanism for working with the components returned by this point-cut. Embedding a framework into an application would consist of creating a point-cut and then passing it to the framework in order to process it.
This would be application-level introspection and reflection and it would do away with both annotation-creep and XML-creep. I can`t dwell too much on it cause I gotta go to sleep.

Later Edit 3: In this post Bill Burke argues about using meta-annotation in order to de-couple your code-base from framework specific annotations. Adding a level of indirection between framework-specific annotations will free your application from framework-specific dependencies, this is for sure, but you will still tie your code to a representation (the representation specified by your annotations that serve as end-points for the original framework specific annotations). I am not sure that this is desired. I still think that your code-base should not be polluted by any external concerns.
It is interesting to see what will happen:
1) Will the world continue with annotation-creep and XML-creep for plugging their applications into various frameworks?
2) Will the world settle for a mid-way solution, like the meta-annotation solution suggested by Bill for the same problem?
3) Will the world try to solve this thru a different design?
4) Will the world decide that frame-work plugging is so costly that we might as well migrate to a different language/platform in order to reduce these costs?
It is all a question of supply and demand basic economics: plugging an application into a framework is currently a pretty costly operation. If the demand for this operation rises significantly (the number of times an application is plugged into a framework increases) it would make sense to make an investment (new methodology, new framework, etc…) in order to slash the costs. Either you make the investment or your language/platform becomes so costly so that it will prove cost-effective to move to a different language/platform.