You have arrived at your 30-day allowance of 5 free premium pages from America’s Finest News Source. If you enjoy our probing and analytical journalism and want full access, we ask that you support our hardworking reporters by purchasing a subscription for as low as $2.95/ month or $29.95/ year.

I was surprised that The Onion would ask you for paid membership after reaching a certain number of articles per month despite the fact that The Economist, New Times, Financial Times and other online newspapers have been using it for a while. A while ago, during the time that paid membership was written off to be replaced with advertising revenues, I alluded to the fact that the disappearance of prices was a temporary phenomenon which simply indicated an imbalance between the supply and demand of news.

The current price signals are indicating a rebalancing of supply and demand, a lot faster that I thought it is possible. The newspapers which managed to go thru this process will emerge with much greater strength. With the competition gone they will be able to operate at a scale which before was hard to imagine in order to become the first truly global newspapers.

P.S. The title is a tongue-in-cheek reference to Web 2.0 and whatever it was supposed to mean.

This is not to say that Apple or Facebook are not profitable enterprises, or that their products are fads approaching their expiry date, only that they are bound by the very field they choose to dominate. Just a feeling…

All the best to the DataSift team, they are building a terrific product!!

However, I don’ think that the solution proposed by Dan Prichett to correcting the IT inefficiencies can be applied easily to any organization. I see the IT architecture driving the structure of the organization, rather than the reverse, happening mostly in IT-centric companies (*). I don’ see an insurance company or a distribution chain changing the way they do business solely for the purpose of enhancing the efficiency of IT. For a typical company IT is a sunk-cost rather than a strategic asset (**), restructuring a company in order to minimize this particular set of costs, possibly at the expense of other concerns, does not make much sense.
Adapting the IT infrastructure to the organization, and not the reverse, will be the norm for the foreseeable future. In most cases the best that IT can do is to try to reduce costs by automation of various tasks across the organization rather than try to define the organization.

Later Edit: Another reason why aligning the organization with the architecture (rather than the opposite) does not apply all the time is that refactoring the architecture will imply re-structuring the organization in order to align it with the new architecture. The already high costs or re-architecting the IT infrastructure may become prohibitive.

* I would call an IT-centric company a company that proposes services built around IT or that derives a  competitive advantage from its IT infrastructure. Examples of such companies are eBay, Amazon, Google, online shops, online media companies, etc… In these type of companies we may derive the organization from the architecture of IT systems because in doing so we enhance the competitive advantage of that company.

** For IT-centric companies this does not apply, in their case IT is a strategic asset.

The article showcases how the Massachussets Department of Transportation decided to open up a bus schedule data feed and found out that within a short period of time a number of client applications started using it. What is really interesting is that these applications provided basically the same functionality across a large number of platforms (iPhone apps, web-site, SMS and IVR systems). Basically MBTA delegated the development of client applications across multiple platforms to parties best suited for doing so.
A pretty neat division of labor emerged: the management of the data feed is carried out by MBTA while the client apps are developed by people at large.

Interesting as it may seem this development is neither new (mash-up have existed for quite a while) nor the beginning of a revolution in IT. This development is just the creation of an eco-system of applications built around MBTA’s data feed. I don’t share MBTA’s enthusiasm over it because delegating the development of client applications to benevolent third parties undermines the reliability of the bus schedules delivery. While this approach did allow them to cover a large number of client platforms without a significant investment it has also created a number of applications which seem to be nothing more that a hobby and which may be taken down without any notice. The eco-system of client apps will evolve over a certain period of time before settling down, during this period the delivery of bus schedules will be as fickle as the applications in charge of it. I am not sure that this is what MBTA had in mind when it opened its data feed to outside parties…

All in all, an interesting experiment in the relationships between various applications, but not a tipping point for the IT industry.

A point that I found very interesting is that systems must communicate via events and not via commands. For a system (master system A) to issue a command on another system (executing system B) it needs to know the context in which the command should be issued on system B. Requiring system A to have knowledge about the inner workings of a remote system in order to be able to carry out its tasks is typically how hard-coupling between systems starts.
The solution outlined by Stefan Norberg is to split the business logic for a specific domain into gathering data required for executing a certain task, transforming it into events and issuing events (perfomed by multiple systems), channeling the events to the system implementing the domain (typically performed by an ESB) and the implementation of the domain (performed by the system which receives the events from the ESB and uses them either for creating domain-specific commands or for creating the context in which domain-specific commands are executed). This partition of tasks scales very nicely as the number of systems and domains grow.

Another point that I found interesting was that this architecture requires the delivery of events in the order they were created. Some events are used for creating the context in which the events which are transformed into commands are running and therefore the events that create the context need to be processed before the events which are transformed into commands.
Let’s give an example. Suppose that the systems involved are a user management system, a bet processing system and a system managing user loyalty. Let’s say that we have this use case: user creates an account and then bets 5000$ on a horse race. For the sake of argument let’s assume that this use case involves the generation and dissemination of 2 events: UserCreatedEvent and UserPlacedBet event. Now let’s suppose that the system managing user loyalty receives the 2 events and is ready to process the UserPlacedBet event by checking the size of the bet and carrying out the appropriate command. Let’s say that for this command to the executed the user must exist in this system’s database. It follows that the UserCreatedEvent should have been processed before the UserPlacedBet event in order to create the context (in this case the context consists of relevant user data) in which the UserPlacedEvent could be turned into a command and executed.

A presentation which covers the use of domain events from a large number of angles (enterprise architecture, management, physical implementation) and which should be watched by anyone interested in large scale distributed systems.

A pretty good picture of the difficult terrain that Europe has to navigate.

Agile methods grew out of the fact that up-front design which involved months of analysis was getting out of synch with the world which required faster development cycles and the need to adapt to a world changing at high speed (*). However, the responsiveness of agile methods and their focus on the next iteration limited development the scope at which development is carried out and eliminated the design phase out of the development process. The result were applications for which development slowed, if not even stopped, once the application reached a certain complexity.

Eric argues that in order to fix this problem we need to bring domain driven design back into the picture, but different from the top-down, process-heavy manner in which it was conducted before. The main reason modelling an application iteratively is that the development of an application is a learning process, a process of discovering and annotating a particular domain.

Some agile techniques such as the ability of reacting swiftly to changes and to be able to perform significant changes late in the game are presented, but they seem to me to be liabilities rather than assets. Performing a significant change late in the game is a very expensive operation since it is essentially a re-organization of the concepts encapsulated in the model followed by the confusion generated by the dissemination of the new model; such things should be avoided at all costs. Rapid changes in the model are also bad since they can create confusion.

One agile practice which I think benefits domain driven design is the break-down of the design process in iterations during which new domain concepts are inserted into the model. It is very important that these iterations also make sure that the model stays in synch with the domain in order to avoid late, massive and expensive domain re-factorings. In order to do this Eric outlines a series of diagnosis measures used for detecting when the model is straying away from the model.
While these measures are necessary what is missing in this process is a way for the domain expert to validate the model. The knowledge transfers have no feed-back loop from the domain experts. The developer seems the sole owner of the model, ideally the domain expert could become more engaged in the definition of the model rather than a passive disseminator of knowledge.

Modelling brand new domains or domains in a state of change will probably always be error-prone because the development of an application in a such a domain is a discovery process for all the parties involved (from users to business analysts to developers) whose end-result is defining the domain. In such an environment probably the only way to define and disseminate domain knowledge is to set-up sessions with domain experts. However, knowledge transfers in mature domains should be handled differently in order to leverage the existing knowledge. It is a good idea to get into the domain by first reading up on it and then engaging in knowledge-transfers with domain experts.

Eric Evans covers a lot of ground in his talk from the methodology of transferring domain knowledge to diagnosing misalignments between the model and the domain, solutions to correcting such misalignments and strategies for coordinating teams working on the same project. However, I wish he would have talked a bit about the structure and skill-sets of the teams involved in domain-driven development, of the ways to disseminate domain knowledge into a team and of how to engage the domain experts in reviewing and providing feed back on the domain being developed.
All in all a very good presentation which I encourage people to watch.

* The rejection of the analysis phase by agilists could partially be explained by the fact that the agile movement started at a point where appeared brand new domains (such as e-commerce, e-marketing, content management) which were in a continous state of transformation and for which there was no prior knowledge. Analysis in this case was viewed, to a certain extent correctly, as a process which requires significant investements (in order to overcome the lack of knowledge) without a clear return (the results could be obsolete in a few months due to unforeseen changes).
At the same time the revolution in communications created the opportunity to add on stakeholders previously considered un-related to the domain. Encapsulating requirements in an application from unfoereseen stakeholders meant that the domains became more prone to change and harder to predict.
These factors contributed to the misconception that analysis hinders rather than helps.

A batch of regression tests covering an appropriate part of an application is a very good strategy to deal with growing complexity. Furthermore, the more complex the application the higher the need for regression tests because as systems become more complex the components that make them up tend to become unwieldy (*). In a heterogeneous team with a high variation in skill sets the regression tests are probably best written and carried out by the most experienced members of the team (it goes without saying that the implementation of some functional aspect and the test covering it should be done by 2 separate persons). If the tests are written by the most experienced team members this renders them more expensive, as a result care should be taken regarding the scope of these tests. I generally consider that regression tests should cover higher-value use-cases at a higher scope and that this scope has implications for the testing infrastructure.

The tests themselves should be isolated from testing infrastructure concerns and should focus mostly on the domain that is being covered. A pretty good discussion of these items can be found here. However, the tests should also be run in an environment as close to the production environment as possible, having seen an instance where tests covering an application deployed on a legacy database were passing without even inserting a single record in the DB only to have the application have problems later in PROD because of legacy datastructures.

Manual regression tests are expensive and this high cost has some bad side effects apart from having a higher price-tag: manual regression tests tend to be executed late in the release process. Also, by the fact that they take longer (being manual) they tend to cut deeper into the safety buffer dedicated to fixing bugs found during the release process. In order to solve this problem the regression tests should be in a format which allows them to be carried out early in the development process in order to detect problems early, ideally as close as possible to the development. This would mandate that the regression tests have a high degree of automation and a certain level of embedment into the development process.

Ideally, automated regression tests should be written in a format that ensures that the tests can be manipulated with ease by both developers (in order to be able to run them at an early stage in the development process) and by domain experts/business analysts (in order to prove their validity). A solution which may satisfy these 2 different audiences could be regression tests written in the programming language of the application by experienced team members and which are read-able by domain experts (**). For complex domains the testing infrastructure may need to be adapted in order to satisfy the second point as well.

* Components become out of synch with the required functionality as the complexity that they need to cover grows. Most of the time this misalignment is due the fact that the development pace is too fast and the information for correctly re-designing components for new functionality is unavailable. As components become unwieldy the need to refactoring appears and with it the need for regression tests which are a key component in the refactoring process.

** In practice, however, it is pretty costly to create a test environment that satisfies these 2 different audiences. This is why automated regression tests are typically written in the language used for development with senior team members acting as domain experts for writing tests of validating tests written by team members.

From my experience there are a few obstacles to using DDD in a typical development environment and one of them is the vagueness which accompanies the definition of different artifacts (entities, value objects, etc…), vagueness which sometimes may generate conflicts (*). The same vagueness works against the size of the development team and the distance between developers (**). DDD is probably best applied in settings which minimize both the potential conflicts and additional overhead of creating and distributing new concepts: small teams of experienced developers in close geographical proximity.
DDD projects tend to exhibit a high degree of closeness between the code and the domain which is a pretty good thing. Code which binds to the domain closely tends to “document” the domain and drive down the time for making knowledge transfers.

Given that DDD is best applied at the high-end spectrum of the IT workforce it follows that it can act as a big differentiator in the productivity of development teams as it further increases the productivity of experienced developers away from the productivity of less experienced developers. However, the gains in productivity may be counter-balanced by the reduced size of the available developer pool and the higher costs and dependencies associated with it.

I would conclude with the fact that the current way of promoting DDD probably works against it. Typically DDD is portrayed as an art-form or craft available to the lucky few which can appreciate and exploit its finesse. This is a pretty good way of driving the typical IT project manager away, IT managers really disliking to rely on artisans rather than on professionals.
For DDD to be applied at a larger scale it needs to be accepted by the business and for this it needs a more “professional” image. Rather than focusing on the “beauty” of various DDD patterns the  DDD community should probably focus a lot more on how to better formulate and deliver DDD best practices to a large audience. This, or it risks to remain at the fringes of software development.

* These conflicts are usually generated when developers cannot agree on a particular design. Given that DDD pushes design right into the core of the software development process these conflicts are bound to appear more often. In order to minimize these conflicts a certain level of experience in software modeling, of communication skills and, to a certain degree, of negotiation skills is needed. All this, in return, requires experienced developers. It may also require to have a domain expert handy (which is typical of DDD projects where the domain expert is a key person in the development process) in order to proof-read the design.

** Size and geographical proximity of teams are determined by the costs of effective communication (costs which usually increase with size and distance). To these communication costs we need to add the costs of debating vague concepts, of defining them and disseminating the knowledge required for manipulating them. These additional communication costs come at the expense of the usual communication costs and require them to be reduced.

If you followed various presentations on Event Driven Architecture for a while you must be familiar with one advantage that many people talk about without going into detail: the ability to recover from crashes simply by re-playing events that were sent to your system. Most presentations give the impression that a flow-based system based on passing messages is born with this ability, but the reality is that it must be designed in order to implement such a functionality.

When designing such a system you first have to ask yourself if you need this ability and I would say that the answer in quite a few cases is yes. The most basic recovery from crashes for a flow-based system consists of the message broker booting up, determining what messages have to be sent and re-sending the messages to the message consumers. Chances are that re-sending the exact messages that caused the crash will cause another crash and in order to avoid this you should be able to wipe out the message store, determine what events need to be re-played and replay them in an orderly fashion till your system goes back to its normal state.

Next, you should determine how to design such a system. One way to design it would be to code the stages as idempotent operations, that is, operations which when carried out multiple times give the same results. However, sometimes the stages of the model of the system are not easily captured in idempotent operations and sometimes it is downright impossible (*).
Another way to design it would be to break the flow into stages and the stages into 2 categories: idempotent stages and non-idempotent stages (**). Next, record the requests that come in and record each stage that a request has completed successfully. For non-idempotent stages also record the state of the request after their completion. Replaying requests in such a system consists of determining what requests are in non-idempotent stages and replaying them from these stages. For example, let’s say that you have a system that accepts orders, performs matchings on them (matches buys to sells), creates fills out of these matches and sends the fills out to an outside application. This system has 4 stages: order receival, order matching, fill creating and fill forwarding. Let’s say that order receival is a non-idempotent operation, order matching is an idempotent operation, fill creation is a non-idempotent operation and fill forwarding is another non-idempotent operation. In order to design such a system for re-play you will need to track a request across all the stages, determine what requests are in non-idempotent stages and in the case you need to replay the requests to replay them from the non-idempotent stages.

Replaying requests could also help you releasing a new version of the system in which the classes of objects which are sent from one stage to another change. Typically when such a release is carried out any message in transit cannot be processed anymore because of class versioning exceptions, adding the option of replaying the requests after updating the flow with the latest release would help solve this problem.

* One example of an operation which may not be able to be made idempotent is sending messages to an external party. For this operation to be idempotent it would be necessary for the external party to be idempotent (that is, it would mean that the same message sent multiple times to the external party would have the same effect). This assumption sometimes turns out to be invalid.

** One example of a idempotent stage is a stage that performs some transformation/computation on the messages it receives and that forwards the messages to another stage. One example of a non-idempotent stage is a stage that persists data to a datastore or sends messages out to a non-idempotent external application. ]]> http://microintellia.com/blog/2010/03/29/replaying-requests-in-flows/feed/ http://microintellia.com/blog/2009/12/17/replication-of-data-and-replication-of-functionality/ http://microintellia.com/blog/2009/12/17/replication-of-data-and-replication-of-functionality/#comments Thu, 17 Dec 2009 17:10:58 +0000 <ADMINNICENAME> http://microintellia.com/blog/?p=476 From my experience I see that replication of data usually helps deal with performance problems, but that sometimes this replication involves replication of functionality, sometimes you end-up replicating some data on a different system (let’s call this the client system) and then you find yourself needing to code on the client system some piece of business logic which resides on the system from where data comes originally from (let’s call this system the master system).

More often than not this is a pretty scenario, because every modification to the business logic done on the master system has to be re-coded on the client system. Sometimes you find that you need to coordinate the releases of these 2 separate systems in order make the whole picture functionally consistent. One way to avoid re-coding the business logic on the client system is to expose the business logic on the master system thru some remoting mechanism (EJBs, Hessian, etc…) and then have the client system pass the data that it replicated locally to the master system in a synchronous call. While this solves the fact having the same business logic reside in 2 places it makes the client system dependent on the master system (*).

One solution to the above conundrum that is to replicate the data, along with the results of carrying out that piece of business logic on that data, to the client system. With this arrangement the client system does not need to have the master system up and running and it also does not need to re-code the same business logic inside. If you manage to use this (this approach doesn’t work in all cases) you will find yourself exporting business logic asynchronously by exporting both data and the results of carrying out that business logic on the data.

If I don’t write another post till January I wish you a Happy New Year!!!!!

* There is a physical dependency (the client systems needs the master system up and running for it to service calls) and possibly a library depedency which may couple the release cycles of these 2 systems (if the client system communicates with the master system thru objects rather than thry wire protocols you will need to synchronize the release cycles of these 2 systems in order to make sure that the client system doesn’t run in PROD with obsolete libraries).

However, doing only this would pose a concurrency problem, namely that two or more transactions batches arrive at the same time when no trade file has been created yet. If each transaction batch would check if the trade file exists concurrently and try to create it, again concurrently, we could end up with duplicate trade files. One way to avoid duplicate trade files is to detect if a trade file needs to get created, allow one of the transaction batches to create the file while blocking the other transaction batches till the trade file gets created. We looked at the costs of blocking and as the costs looked pretty small (we would be blocking only once time per day when the file gets created) we decided to go ahead with blocking.

However, this approach clearly doesn’t scale, and we implemented it because it the conditions for blocking happen very rarely (as I was saying once per day) and it would not be feasible in the case of a higher amount of contention. We looked at some non-blocking alternatives and it looks like a good one would be to allow the transactions to check if trade file exists and if not then to create the trade file on the fly (without blocking) and at the end of transaction processing send further a message saying that there is a risk that some data is inconsistent (namely that there is the risk that some files have duplicates and transactions are assigned to duplicate files) and establish a procedure for repairing the transactions (if necessary). This would allow for non-blocking flows and higher thru-put, but it would come at the expense of a period of time in which data is inconsistent (in our case there is the risk that some transactions will be assigned to duplicate trade files till the duplicate trade files get fixed).

If inconsistent data is OK for the business and the rest of the application (it could be that these repair procedures as well as inconsistent data affect other parts of the application) and if blocking flows are creating significant performance problems then allowing for data to be inconsistent for a certain period of time while providing a mechanism for detection and repair of inconsistencies would probably solve the problem.

Another solution to this problem would be to detect messages which may cause blocking and create a new stage in the flow which deals with such messages.

P.S. I was shocked to see the date on that article: 1971. The fact that we are still struggling with these problems shows a pretty big problem with computer science. Most people equate computer science with exotic algorithms, it would be about time to add to this equation software modelling. It is pathetic when useful concepts for writing software developed 30 years ago are still unknown to the typical developer.
Software development has a pretty big educational problem (most developers educate themselves on their own and learn on the job), it would not be a bad idea to address it.

If this happens it could happen that VMWare will become the premier hosting service for cloud-deployed enterprise Java applications and for specialized, Java-powered, cloud-deployed enterprise services. If the cloud is the future of application hosting (frankly, I would not bet the farm on it) then VMWare will be the hub where Java applications will be hosted (the difference between hosting a Spring application with VMWare and with a different cloud provider would be the difference between a virtual machine specialized for running Spring applications and an all-purpose virtual machine. This difference in the efficiency of hosting a Spring application would drive more users to VMWare’s clouds).

Still, 420 million dollars is a pretty big sum of money, especially in the current economic climate, and we will have to see if VMWare and SpringSource will find the synergies which will make them pairing together more than the sum of their parts.

Congratulations to Rod Johnson, Adrian Colyer, Juergen Hoeller and the rest of the Spring team and all the best to VMWare and SpringSource!!

Later edit: It looks like you can buy Spring-images on Amazon’s Cloud thru a company called Cloud Foundry which was acquired by SpringSource. I am still trying to figure out why VMWare bought Spring Source rather than entering a licensing agreement with them. I guess we have to wait and see.

Later edit: According to Adrian Colyer this is the future of how Java applications will be deployed. Still, I don’t buy into the cloud hype, I think that for the near future cloud adoption will be determined mostly by prices for different pieces of hardware. Maybe at one point there will be a difference in the efficiency of managing applications deployed in a cloud versus the ones deployed outside which would compensate for the differences in price of hosting an instance locally or on the cloud, but I don’t think we are near this point.

The copyright concept is tied to the ability to make copies: when this ability did not exist (such as prior to the phonograph) or was severely impaired (such as during the vinyl era) the concept of copyright did not exist because there was no need for it. Copyright appeared when music recording moved to a medium which allowed for music to be copied easily enough for music buyers to make their own copies (roughly around the introduction of the compact cassette), hence the introduction of the right to copy. In the current environment characterized by very low copy costs and very inefficient means of enforcing rights to owning a copy you could say that the concept of copyright is living its last moments.

Copyright may appear to future historians as a concept which lasted for a period of time during which the ability to copy content was both developed enough to worry content creators and distributors about people getting content without paying for it and at the same time it was under-developed enough to allow the content creators and distributors to develop mechanisms for controlling content copies. Once the ability to copy and share content became  utiquitous the right to a copy of content ceased to exist as the means of enforcing this right started to weaken.

The question that is posed quite frequently is what will happen to musicians since they will not be able to profit from copyright due anymore? I would say that you could compare their fate to the fate of musicians before the concept of copyright started to exist; which means looking at music before the introduction of the compact cassette ( whose low copying costs allowed for easier manufacture of copies, which in turn allowed for greater diversity in music): musicians would either attain the status of Enrico Caruso and make their living by selling music copies or would rely on gigs building a more or less devoted following. The main difference between a musician relying on gigs in 1910 and 2010 would be that the musician living in the 2010’s would act at a global scale (in terms of distributing and marketing his music, of setting up a tour, of building a fanbase, etc…) while the musician living in the 1910’s would probably have acted at a local scale.

I think that the main difference between the old media and the new media is that the old media is delivered thru a vastly more narrow channel than the new media, and that the width of this delivery channel pretty much dictates the format and the news that reach you.

Your typical daily is a newspaper with a limited number of pages. The number of pages is both large enough so that it allows the daily to cover more sections, but at the same time small enough to keep the number of sections in check. A typical daily ends up having Local, National, International, Business and Sports sections.
The limited number of pages means that the stories are competing with each other in order to get printed. The result is that a daily covers only the issues which are the most important to its readership. Some print dailies try to compensate for the limited amount of articles that they make available by the quality of these articles (this partly supports the case of print media moving into the luxury goods category because of the scarcity of items its format can deliver when compared with new media).
The limited number of pages also means that there is competition between the journalists wanting to  get published, and this further enhances the quality of the news paper, typically print media which has a large supply  of journalists (New York Times, Washington Post, Los Angeles Times, etc…) tends to print articles of very good quality as only the best journalists get published.

New media changes all this: the very wide channel along which content gets pushed to readers increases the diversity of the content and while it dimishes its quality. The only constraint that I see on content consumption in new media is not the scarcity of supplied content, but rather the opportunity cost which comes with consuming a particular piece of content: when you are consuming some content you are forgoing consuming some other content.
I think that opportunity cost will define the way content gets consumed in the new media and that the organizations operating in the new media will have to dedicate a bigger amount of resources to marketing themselves because the competition between new media organizations will be far fiercer as the typical boundaries between content sources disappear (*).

* One example of a boundary between content sources which disappears and generates competition is spatial distribution of news papers: New York Times and Los Angeles Times which were not competing with one another previously (each being distributed in different cities they were competing only with local newspapers)  are now competing with each other when they are both distributed via the web.

One type of alternative datastore that is getting a lot of attention is the distributed key-value store which maps values to keys and then assign keys onto multiple storage nods according to a hash function (**). In such a setting you get an object thru a key, you work with it and then you save it back in the datastore. The fact that this configuration scales out easily (***) makes this datastore very appealing, and if you are working with large datasets you will probably have no choice but to use something like it.

The transition from relational databases to key-value stores will probably include taking relations out of your application, re-modelling your application in order to group data together and streaming data to a reporting database. Relational databases gave the user both data storage as well as relations between entities which came as both a blessing and a curse (while relations made reporting easier they also gave un-checked access to data which allowed all sorts of corners to be cut). Well, key-value stores will take relations out: data access is more local, you typically have access only to what is immediate to a particular entity and you cannot perform queries spanning your entire data model. This type of data access could turn out to be actually a good thing, because these very close relationships will force an application to have a cleaner design since you will not be able to rely on monster SQL statements for compensating for design deficiencies.

In an application which needs to handle massive amounts of data relations between data will be delegated to activities which require them (reporting is a pretty good example) and data will be streamed from the key-value stores to the relational databases where these activities take place.
All in all, a pretty good division of tasks: applications with a (hopefully) cleaner model relying on key-value stores streaming data to relational databases for reporting.

All the above doesn’t mean that relational databases will disappear, the vast majority of applications do not require to process the massive amounts of data which render your typical database unpractical. Relational databases will be around for quite a while.

* Obviously, there are database clusters (such as Oracle RACs), but they are pretty costly.

** Examples of key-value stores are Google’s Big Table, Amazon’s Dynamo and IBM’s eXtreme scale.

*** Actually, there are constraints attached to it. In order to get the best performance you need to model your application so that data access occurs within the same node in order to avoid a distributed transaction spanning across multiple nodes. Please read this article by Billy Newport from IBM for a better understanding.

I think that Chris Anderson also makes a mistake when believing that the main costs which define how information is consumed are the costs of distributing information, distributional costs are just one part of the story, one other important cost would be the cost generating the content. Given that in the current period the distributional costs are very low, it would follow that the costs of generating the content would be the bulk of the costs of consuming information (i.e. the cost of consuming information would be identical or very close to the costs of creating the content). Basically what happens right now is that the supply for content is dwarfing the demand for it, we are simply swamped in content. Well, this mismatch between supply and demand drives down the revenues of content sources. In addition to this,  one side-effect of low distribution costs is that various content sources are put in pure and un-distorted competition and it is this competition that is driving even lower the revenues of various content providers.
It appears that the content sources that are not experiencing reduced revenues are the ones for which the content supply is very small when compared to demand (and which creates the scarcity which could allow a content source to charge for usage) and which are not experiencing heavy competition (some financial newspapers being a very good example). If anything these content sources have found in the web a new channel thru which to distributed their small supply of content. I would say that content sources which exhibit both a small supply (small enough to create scarcity), a reasonable demand and not a lot of competition will be the content sources which will be economically viable in the coming years (*).

The near future will be very challenging for the traditional newspapers, because the tremendous amount of excess capacity in this space will have to be reduced thru various means (**). Once reduced, the remaining capacity will probably be small enough to match the demand for the content it produces, and at that point various revenue models will appear. And charging for usage will be part of those revenue models, regardless of what various luminaries think about it.

To sum this post up I would say that the newspapers are in a pretty currently bad corner: the new low distribution costs have put them in fierce competition which resulted in massive excess capacity (***). The excess capacity will be dealt with one way or another and at the end of this process the remaining newspapers will come back to health.

Later edit: If you have read enough material on the changes currently affecting newspapers it is very likely that you have come across the concept of “economics of abundance”, a term used in science-fiction before the Web2.0 “revolution”. In the context of current changes in media “economics of abundance” is just another synonim of over-capacity.

* Incidentally, niche content sources fit this description pretty well.

** The excess capacity could be reduced by differentiation. One example would be New York Times rising the price for its street paper, which some argue that it pushes NYT into the category of luxury goods, differentiating it from the other news papers. Specializing in a new field is another way to direct excess capacity towards more productive uses.

*** The increased competition between newspapers comes from the fact that while previously the competition was mostly local between local newspapers because content distribution was mostly local in the current period the distribution is global, therefore more newspapers, from any conceivable place, are now competing for the same pairs of eyeballs. Right now the same news item can reach you thru a very large number of channels, while previously it could reach you only thru a few.
I don’t include the dozens of millions or so of blogs into the excess capacity mentioned above because blogs are not in the same league as newspapers, the differences in quality between a typical blog and a newspaper separate these 2 types of content.

All this aside, it is a pretty interesting proposition from Google and leaves you wondering about how would a group of persons that communicate with one another by using the translator package just added to GMail outside GMail, let’s say in a face-to-face environment (let’s say a group of experts from various places , all speaking a different language attending the same conference). Would they be emailing each other while seated at the same table, given that they do not speak a common language (*)?
A pretty weird feeling is setting in: you may end up having relationships with people only thru this virtual medium which negates all other forms of communication. The language barrier will be replaced by an expressiveness barrier: many things will not be able to be expressed thru this medium and will be left-out.

* BTW, the message used in the TechCrunch example is a very good example of a message that would never get translated because it would never be written in a different language: if you want to invite a hiking buddy out for a walk you would probably want to be able to communicate with him/her in a commonlanguage, not thru the GMail translator… I wonder what a walk thru a park is like when you need to drag along GMail (preferably deployed on your iPhone) in order to talk to someone. Surreal…

There is one big difference between ordinary desktops and mobile phones, though, the ergonomics. As the feature set of an application increases so increase the demand for a larger interface in order to interact with that richer feature set (which could be deployed remotely, BTW). This would mean that third world consumers will first interact with an application thru a cheap mobile phone, and access a reduced feature set, and as they grow richer they could afford a tool which would allow them to access the same application thru a different, more expensive and graphically more expressive interface (the same application accessed thru a netbook).

The difference in income would drive the difference in access to the very same application. This could allow for identifying customers with higher incomes by the way they access the same free resources…

I will get into an example where using DSLs is probably not a good fit because of organizational issues rather than inner strengths and weaknesses. Let’s say that you have some distributed teams working on the same application and that these distributed teams are led by a team of architects which need to create an environment in which these distributed teams work together. I find it pretty hard to see how this environment could be constructed using DSLs, particularly iteratively-developed DSLs, because changes in the DSL would have to be propagated across multiple teams in dispersed locations which would need to be re-trained for every iteration. Compare this approach with creating an API that would be released every so often and for which re-training costs are very small.

I have the impression that the main cost of DSLs is the distance between the place where the DSL is produced and the place where the DSL is consumed and that DSLs tend to be either specific to the location where they are produced or widely known and used within a community (in the second case the distance between the DSL producer and the DSL consumer is compensated by “mind-share”). I would say that the main difference between a DSL and an API is that DSLs are more tightly bound to the domain (by definition) and that part of the costs of using a DSL is the cost of transferring the knowledge about that domain between various stake-holders. This would mean that domain experts can adapt to a DSL targetting their domain faster than a typical developer that is new to the domain and that domain experts would be more productive. The difference in productivity between various members of the work-force means that the work-force composition should partly drive the decision to choose a DSL or APIs on a particular project. Regarding the distance between DSL production and DSL consumption this paper discussing how distance affects knowledge transfers is probably a good read.

Other issues wih DSLs such as developer lock-in (a developer working with a DSL can become out of synch with the rest of the IT landscape), the lack of language design skills on the IT job market, the problem of capturing domain knowledge correctly in large or rapidly changing domains are not explored in this post.

I end this post hurriedly…

Later Edit: I was watching this interview with Charles Simonyi and I would say I remained with these impressions:
If the DSL comes from an already existing language, such as domain experts notations, then there are chances that the DSL will be a good language (this addresses the issue of lack of language design skills on the IT job market). The problem of creating a new language is done away with by re-using a currently used language (the domain experts notations) and creating a new way to implement this existing language.
Regarding domains which are changing rapidly Charles Simonyi says that by separating the notation from the actual domain schema it becomes pretty easy to change the notation (the DSL) without affecting the underlying domain schema. Creating a DSL becomes an iterative process.
End-user programming is to remain a pipe-dream because of the difference in skills between what is needed for programming and the difference in skills needed for working within the domain. DSLs seem to engage the domain expert more into the process of software creation rather than turn domain experts into programmers. Domain experts contributions to software creation thru DSLs may be simply to communicate the domain better to a programmer. Language work benches would create a new division of tasks according to skills.

LLE: I watched this presentation by Intentional Software on domain work-benches and I remained with these impressions:
In an environment where domain workbenches are used it looks like the programmer will be involved in creating projections to an operational environment. The developer will focus on the systems while the  domain experts will focus on the business logic. This division of tasks addresses the issues arising from developers’ lack of enthusiasm about the domain.
It looks like domain work-benches should be used in large and rapidly changing domains where the transfer of knowledge from domain experts to developers is very costly due to the size of the domain and the speed at which it moves (this answers one of my questions above).

I think that the argument that I was making above that DSLs will be location-specific still applies even with a domain workbench in the picture because knowledge transfers with high costs will not appear in the creation of the DSL itself (the DSL being just one notation for an underlying domain schema, one notation among many, it can be created and used locally) but rather in the creation of the domain schema itself (which may end up being a collaborative effort carried out among people in various locations).

As I was saying above I have the impression that in the near future domain workbenches and DSLs will be used where the costs of knowledge transfers from domain experts to developers is large. If the costs of creating languages start to drop and these domain work-benches become ubiquitous then it is possible that a complete separation between domain logic (stored in domain code and expressed thru various notations) and its implementation in a system (which appears to be essentially a projection of the domain code into an operational environment) will appear. It would be interesting to work with Intentional Software’s domain workbench too.

We will have to wait and see…

L3E: This is another presentation on the Domain Workbench.

One pretty good example of such an application is Technorati, an application feeding off the blog frenzy that characterized the mid 2000’s. For a few years it was the go-to site for everything blog-related, today it is a has-been as interest in blogs was replaced by interest in social networks and Twitter.
I see one small problem with applications which follow and try to satisfy consumer trends: the fact that they require large investments in infrastructure (demanded by the large loads they have to satisfy) as well as the fact that they need to re-coup this investment in a relative short period of time before they become fads. 

I am wondering how volatility in user tastes will affect the funding and the revenue models for applications which are trying to satisfy consumer trends, because the gap between the investments necessary for putting up such applications and returns on these investments grows larger as the volatility increases. VCs will not be able to hedge away risks by investing in multiple applications at the same time (effectively spreading risk across different applications in the same class of applications), because all these applications could suffer large swings in their userbase. VCs will probably have to hedge away the risks of investing in these applications by investing in areas outside social-networks (green technologies, etc…).

User volatility is a pretty new phenomenon for which there is no history from which to infer much. It will be interesting to see how this will un-fold: will this type of application simply disappear because it is economically irrational to invest in them or will the costs of running such an application become so low that the investment required for it can be amortized in a short period of time? Or something else…

Reading the press release I came across this worrisome snippet: “Sun is a pioneer in enterprise computing, and this combination recognizes the innovation and customer success the company has achieved. Our largest customers have been asking us to step up to a broader role to reduce complexity, risk and cost by delivering a highly optimized stack based on standards,” said Oracle President Charles Phillips.
I had the opportunity to deal with a highly optimized stack from Oracle - JDev integrated with Oracle DB - and I know that I don’t want to deal with anything similar to it again, from what I remember developing on JDev means basically being taken hostage to a set of capabilities.

I think that this deal is best defined as the acquisition of the largest open source company in the world, Sun managing these OS assets: MySQL, Java, Open-Solaris, Open-Office,  Open-Sparc. Oracle should treat these projects and the communities built around them with care and not antagonize them because quite a few competitors would love to be put in a position to efficiently fork Sun’s OS assets in order to acquire a new community and a new customer base.

Another point I would like to make is that Oracle is buying Sun when Sun’s grip over the Java community is waning away. As outlined in a post by Rod Johnson development in Java is moving away from EJB containers to specialized solutions which means that the EJB assets bought by Oracle are diminishing their worth.

I think that this transaction will define Java for the near future. Sun did an outstanding job of building a community around Java, a community which delivered an impressive stack of libraries, an open community which could work together in order to avoid duplication, a community which turned R&D projects into viable commercial projects. It would be a shame if this changes in the future.

Later Edit: mySQL just got forked, I thought I may add a “Fork Watch” for the Sun OS products which Oracle has just acquired. It would be interesting to watch how this will unfold…

• First fork of mySQL by the Open Database Alliance.

All in all, a very good presentation.

Let me explain: with the introduction of ubiquitous connectivity partnerships between various organisations became feasible to the point where applications are becoming applications composed of components, components which are not all originating from the same organization. Organizations are finding it wasy to exchange data and functionality (think about the widgets on Facebook: Facebook could import Digg-like functionality and data straight from Digg). The result is that web applications are becoming places where organizations meet, agree on the interactions between them and then implement these interactions. Once a web-app manages to get a certain number of users it has the opportunity to become a portal by swapping data with other web-apps and integrating with these web-apps.

Now, since the place where applications are getting composed is typically the place where applications are deployed, it follows that the place where applications are deployed defines the costs of application composition. If the applications are deployed remotely then the task of composing these applications is shifted onto the teams which are managing this remote application. If the application is deployed locally (on the desktop for example) the task of application composition is shifted onto the user that manages it on its desktop. If the costs of application compositions are low then the task of application composition can be shifted onto the user, if they are high then the same task will be shifted onto the teams that are managing the application remotely.
I would say that application composition costs are application discovery, integration between applications and integration between applications and the host environment and currently these costs are higher for RIAs than for typical web-apps.

Given that the costs of application compositions on the desktop are currently high, predominantly because of lack of standards, it follows that most of the application that get composed will be deployed remotely. Flash/Java FX/Silverlight will probably run stand alone applications which have a low degree of interaction with other applications. Facebook will probably be the place where a lot of the application composition will occur.

The near future is on the web and on web-applications. RIAs will probably satisfy some niche, but will not become the dominant architectural style in the near future, unless the makers of the RIA runtimes (Flash/Java FX/Silverlight) will find a way to address the costs of application composition. Some sort of a container which allows components originating from multiple sources to integrate among each other easily could lower these costs, taking the appeal of RIAs beyond its immediate differentiators (greater interaction with the user, low response times, etc…). Honestly, I am skeptical that the costs of composing application in RIA environments will become low enough to the point where RIAs will become more like portals where interactions between multiple organizations are carried out and less like applications.

Spring source is an a-typical open-source company in the sense that they broke away from creating open source software and moving into creating open source domains. Initially they became known for their IoC container and a few libraries which solved various code issues (their JDBC libraries being one such example). Next, they added various features to the Spring container (proxy-based AOP, etc…) and added some more frameworks to their portfolio, such as Spring MVC, following the typical road of an OS company that senses industry-wide needs and addresses them by creating frameworks implementing those needs.
Lately, however, Spring Source has been diversifying into business domains: right now its product portfolio encompasses Spring Security (actually an old framework known as Acegi Security), Spring Batch, Spring Integration, etc… It is a pretty interesting development and, as far as I know, it is singular in the OS space: we have an OS company that senses domains that are needed across the IT industry, pools together domain experts and goes on to deliver a framework that covers the main concepts of the domain and is sufficiently flexible to allow for covering corner-cases easily.

One interesting point about OS domains is that the company that provides a framework which implements such a domain pretty much needs to acquire a significant marketshare and be guaranteed not to lose that marketshare and actually move to a monopolistic position as long as it delivers a good implementation of the domain. The reason for that is a domain is pretty much a form of knowledge and implementing a domain is essentially translating this knowledge into a particular programming language. If the knowledge has been translated into a programming language successfully (i.e. when the domain has been implemented robustly) there is no need for a competing implementation, because this competing implementation would simply re-translate what was previously translated, which is pointless (the end result is the same - a new translation of the same concepts).
Another interesting point is that the company servicing the OS domain will probably derive revenues from domain knowledge rather than from the framework/application implementing that domain, the support for the framework implementing a domain is actually a proxy for domain knowledge (people buying support for the framework will buy it for understanding the domain better and for using the framework better within the domain). 

If I were to say what qualities are necessary for a domain to become an OS domain I woud point out that such a domain needs to be quite static, it has to be mature enough and it needs to be in great demand, it has to cut across the whole IT industry.
If I were to point a domain which is ripe for becoming an OS domain I would say that partitioning an application to run across a large number of nodes will probably become a good candidate for an OS domain as the number of applications which need to scale out massively will increase.

From what I see Wordpress has a bit of a problem maintaining some coherence between its releases, when I first saw WP 2.7 I first thought that it is a completely new product with a completely new functionality and user base. Time to focus on continuity when designing user experiences at WP…

* I don’t realy use this blog as a blog, but mostly for storing some information that I find interesting. It contains links to my del.icio.us account, etc… I keep it with Wordpress in blog format because it is the most convenient.

At the same time I have found myself growing dissatisfied with the RSS feeds that I use for getting news and articles online and I think that the main reason for this dissatisfaction is the specialization of the content sources behind these RSS feeds. I am subscribed to a variety of content: from articles on software architecture, to financial news, to articles on design, economic development, economics in general, etc… These content sources are targetting a domain and are producing content according to that domain’s needs. One side effect is that by consuming the content provided by these sources you are getting a certain degree of specialization in the domain covered by them (which is very beneficial if you want to acquire a certain amount of specialization in a particular domain). One other side effect is that the specialization of these content sources prevents different content from getting to you and you are becoming hostage to some predefined content.

I find that by getting articles only thru my RSS feeds I am getting isolated from other parts of the world. In part for addressing this I have started reading newspapers again, because newspapers provide that general view on the world due to their format: the content of a newspaper had to be delivered as a whole rather than to be split into multiple slices which could be delivered to the reader independently. This format pretty much meant that if you needed to address new interests for your readership you needed to add new sections to it and as a result to expose your readership to new fields. The old newspaper is a home to multiple fields (for example your typical NYT covers everything from world news, to metropolitan news, to arts, business while also providing some comentary) with a news reader that is exposed to a variety of fields and an editorial team that understands the typical news reader and can provide the appropriate content for it.

This, however, changes a lot in the new media: the newspaper in the new media is getting decomposed into finer and finer slices which are being marketed to the newsreader thru various channels. At the same time the task of choosing what to read is being shifted from the editorial team to the reader which results in the abandoning of large, uniform reader bases and the creation of specialized niches.

If I were to set some resolutions for 2009 I would take a look at the fields into which I want to get some degree of specialization, keep the appropriate RSS feeds and remove the rest. While continuing to read newspapers, of course.

Happy New Year to everyone!!!!

* I recycle pretty much all that can be recycled in a home.

** I get too absorbed when reading news or articles on a computer so that when my kids want to play I cannot let go of the laptop easily, this is the main reason for which I pretty much stopped using a computer at home. When I am reading a paper I just put it down and go to play and I pick up the paper later.

Code reuse granularity goes against the number of the participants because implementing code reuse means deciding on what is common (and prone to be re-used) and what is specific as well as on the mechanisms used for implementing the relationships between what is common and what is specific (via sub-classing, various design patterns, etc…). This process of decision starts to blur once the number of participants increases and the corresponding communication costs start to rise. Too many people arguing about what is common to all of them will have quite a few problems figuring out what they are agreeing on.

The interests of pool of participants also are also more prone to diverge as their number increases, further reducing what is common to these participants. The end result is that once the number of participants increases both the commonalities between them shrink (as their specific interests diverge) and finding and implementing these commonalities becomes more expensive (because of the higher communication costs which occur between a larger number of participants). In these situations the code which gets reused starts to boil down to the main architecture classes within which the participants to the projects are coding their specific classes.

Keeping the number of participants low (agile techniques suggest keeping the number of participants below 7) is a way to deal with a codebase and a way to encourage code reuse efficiently.
However, sometimes you may end-up with a high number of participants. In this case code reuse should be brought down to re-using the main architecture classes in order to avoid confusion born out of mis-communication.
Another technique to deal with a large number of participants is to partition the code-base in order to minimize the number of people which are sharing code, the less people are sharing code the less the communication costs and the more things they will have in common.
Big, unpartitioned codebases come with big problems from what I have seen so far.

Really funny that an online bookstore defines the web and not the Yahoo-Microsoft-Google group. I think that Amazon is ahead of the pack because being an online retailer its competitive advantage doesn’t lie with hoarding data or its super-efficient data-centers and as a result it can make them available to the outside world without damaging its competitive advantage. One interesting difference between Amazon and the YMG pack is the rate of success of each group’s offerings: Amazon Cloud has been a tremendous success, the MTurk seems pretty busy while Google has been releasing one service without customers after another (anyone remembers Google Health?), Microsoft is paying people to use live.com and Yahoo is struggling to figure out what it needs to do next.

This retailer gets the online world and will probably define important parts of its future. The fact that its revenues are indirectly derived from the web will probably make it avoid various conflicts of interests (hoarding users’ data rather than giving users control over it being one such conflict) with which the YMG group is struggling.

Against this background I have read 2 interesting articles on SOA: one article by Martin Fowler about using open-source techniques in order to implement collaboration between a consumer service and a supplier service and one article about abandoning the mindset which focuses on IT and rather on focusing on the relationships between various services thru contracts and policies (contracts and policies are organizational issues and not technology issues).

I found Martin Fowler’s article refreshing, even though I have to say that I disagree with it. From what I understand from the article. I think that having developers commit code to a service that you need to connect to has high interaction costs (basically the developers need to understand the partner service, need to be coordinated with the rest of developers, etc…), interaction costs which drive down the number of services with which you can partner effectively. Martin Fowler takes the issue of partnering with a service even further by having developers from the service consumer become full rights committers to the codebase of the partner service. When your developers are committers to a different service you have to wonder on what service are these developers working. The service consumers and the service providers are becoming very tightly coupled, not thru the interfaces they use for communicating (REST, WS-*, wire protocols, etc…) but thru the developers that these 2 services are sharing. I would say that Martin Fowler open-source like approach to service interactions does not scale to a larger number of services. I could go a bit into the differences between open-source projects and your typical enterprise application (I think that the most important one is that on an open source project features are chosen by boiling down competing features to the lowest common denominator chosen by the OS project participants while your typical enterprise application chooses its features in order to boost its revenues), differences which will pretty much tell why OS-like projects are very rare in the enterprise computing world but I don’t really have the time.

I have to say that when comparing the above OS approach to SOA to the typical vendor approach to  SOA you see big differences: a typical vendor assumes that services are out there in a registry and that a business analyst can create new processes and new value by drag-and-dropping various services into an IDE storing references to firm-wide services. This vision does not have much in common with the real world for two important reasons: 1) there are interactions costs associated with consuming each service, interaction costs which are not expressed in a service IDE and 2) in order to manipulate services at a firm-wide scope you need to master knowledge at the same scope. Possessing knowledge at this scope is nearly impossible, I personally have seen rather the reverse: business analysts getting so specialized that their scope becomes more and more focused. As far as I see it in order to achieve the SOA-nirvana championed by your typical IT vendor you need significant disruptions in your work-force.
And this brings me to the MWD article which echoes some thoughts that I wrote a while back when I was saying that interactions between IT services will be driven by the interations between the BUs implementing these services. SOA is not about empowering some business analyst to draw some funny diagrams on a board and re-engineer information flows at the firm level, but rather about how you create relationships and interactions, how you create incentives for adhering to rule and penalties for straying away from them, how you delegate managing an interaction to the parties involved in it, how you identify and consolidate common tasks into shared services and how you allocate tasks which are particular to a small set of parties, how you deal with who loses in these shifts, etc…

As you can see, all these concerns don’t have much to do with IT, being first and foremost management concerns. The only thing that IT can do is help these management issue get resolved, but not resolve them. IT brings some capabilities which can be used: service registries (which consolidate the view of and access to IT assets), ESBs (which allow for controlling data flows), policy standards (which allow the automation of defining and enforcing policies), etc… These capabilities should not be mistaken for their usage or their ends.
Ultimately, all the IT issues on SOA will be pushed to the back and management will take over. Ideally this field will become so specialized and its audience so narrow that we will get rid of another tiresome buzzword…

Later Edit: As you can guess this post has been written in about half a dozen different sessions, each sessions adding a few more lines to what was written before. I don’t have much time to write anymore (and I guess it shows), but I felt compelled to comment on those 2 articles because I thought they stand out from the rest of articles written on SOA.

I would follow this by saying that a lot of time will be saved if the interactions between various parties will be broken down into tasks and that the tasks will be allocated according to the nature of the organization(s) within which these interactions will be carried out. This applies not only message passing but to any other environment in which various entities engage in collaboration. 
For example, you may find out that DSLs will not work in particular setting not because DSLs are bad, but because the organization within which these DSLs are implemented and used simply doesn’t warrant their success.

One of the best interpretations of the differences in REST vs WS that I have seen is the one that Mark makes in this presentation and that I will try to reproduce below is the division of tasks of mapping a message sent to an entity (could be a service, could be an URL) and the component/processor within that entity that will process it: when you send a message to a specific interface you will be specifying the method and the component that will process that message, so the message-to-processor mapping is done by the client. Whereas when you send a message to a specific interface you need to attach some meta-data to that message which will be used by the entity implementing the specific interface for mapping that message to its appropriate processor, therefore the message-to-processor mapping is done by the entity.

The message-to-processor mapping is externalized in the case of specific interfaces while the uniform interface internalizes it. These different mappings come with different transaction costs: in the case of the external mapping the costs are synchronizing the client with changes in the specific interface while in the case of the internal mapping the costs are adapting the uniform interface’s mapper to changes in the meta-data (a new type of messages should be mapped to a new processor).
Now, if we apply economics to the task of message-to-processor mapping we would start to get some interesting results. According to Coase’s theorem you can externalize an activity if the transaction costs for that carrying that activity are low enough and if the transaction costs are high then that activity is better internalized. It would follow that when the transaction costs of this mapping are low you can externalize it via a specific interface, otherwise you would be better off to internalize it via an uniform interface. 
The transaction costs of the specific interface are the synchronization of distributed clients and this could be resolved either thru versioning, or by not modifying the specific interface or by standards. If you can keep these costs down it follows that you can delegate the message-to-processor mapping efficiently to the client else you are better off with a uniform interface. Keeping these costs down also implies that the number of methods that get exposed is kept down in order to avoid the number of changes (more methods, more opportunities for changing them). Generally speaking interaction via specific interfaces is best limited to a few coarse-grained components, between a few participants and with components which do not change very often.
The above synchronization costs run counter to scale in number of clients: as the number of clients starts to grow you are starting to have problems synchronizing all of them and eventually the transaction costs of using the specific interface will become so high, it will not make sense to use it. One corollary of this is that the uniform interface is far better at the scale of the internet and this is probably the main reason why it is used at this scale.
The above will provide a pretty good guide to an architect trying to figure out whether to use a specific interface or an uniform one because their use is not driven by technology, but rather by the organization(s) within which they are employed.

Another interesting point in his presentation was the effort for creating a REST-ful standard for carrying out distributed transactions and the need for agreement between multiple parties. As I was saying in a previous post about REST and WS the differences between these 2 camps are more of organizational nature than anything else. Mark touched upon the fact that even if the need for carrying out distributed transactions via REST exists there is still no standard for doing so, that the REST community seems pretty much happy with implementing low-level infrastructure and not with higher-level, and higher-value, components. I find this a pity and I have the impression that this need will be fulfilled only it raises to a level at which major corporations will find profitable for getting involved. The REST community appears simply way too fragmented to be able to carry out this effort on its own…

Anyway, go ahead and watch the presentation, it is the most thoughtful presentation on distributed computing that I have seen in a while.

It is at the transformation stage that the thorniest issues arise with ESBs. As outlined in this presentation from Thoughworks there is a tendency to push business logic inside the ESB resulting in a very tight coupling between applications and the ESB (part of the business logic will be stored in applications, part of it in the ESB and the business logic in the ESB will need to be kept in sync with the business logic in the applications). Not a nice scenario and unfortunately the solution provided in the presentation (apparently you only need to use Guerilla SOA) doesn’t say much. The business logic which gets shoved into the ESB in the above example are the relationships between the various parties which are sending each other messages and the mess that comes out of it as these relationships change over time.

One example of ESB usage is a customer-facing web application which takes purchase orders from customers and sends messages to an inventory and shipping system, to the accounts receivables system and to the customer loyalty system. There are a few ways that this messages can be sent: 1) the web-application sends only one message to the ESB which then creates 3 more messages, one for the inventory system, one for the accounting system and one for the customer loyalty system (in which case the relationships between the web-app and the 3 systems are managed within the ESB); 2) the web-application creates 3 messages which are all sent independently to the 3 systems (in which case the relationship between the web-application and these 3 systems is managed by the web-app typically thru dedicated senders which are transforming the message according to the other party’s specs) or 3) the web-app publishes the message on the ESB which forwards it as-is to the receiving systems which are receiving the message as it has been created by the web-app and transform it according to their rules (in which case the receiving systems are managing the relationships with the web-application typically thru dedicated receiving points which are transforming the incoming messages).
So far I have only seen 2 or 3 used and it was used successfully. I have not seen 1 being used successfully, but rather I have seen ESBs simply delegated to the role of a dumb-pipe while the processing is done on the end-points. I think that the main reason for which the ESB cannot grow beyond the role of a dumb-pipe is that the transformations typically involve data, imagine that in the above scenario the IDs of the products on the web-application are different from the ones in the inventory system (a very common case BTW). In order to service this transformation you would need an up-to-date mapping between all relationships that it is managing. Sooner or later that ESB will turn into a giant vacuum cleaner that will need to suck every byte of data available in order to service the relationships stored in it.
Lesson #1: Managing relationships typically involves managing data.

Let’s go more into the gory details and think about how we would deal with stale data: let’s say that the datastore holding the mappings of product IDs from the webapp to the inventory systems is missing a product ID (another very common case). In the case where the relationships are maintained at the connection end-points (message producers or consumers) the operation team managing those end-points would have access to the exception (missing product ID mapping in this case) as well as to its solutions (create this mapping in your mapping store). How will the ESB deal with stale data? Well, the ESB will have to find the entity which can resolve this exception, package the exception and send it to that entity. Sooner or later the ESB gets becomes a spaghetti bowl.
Lesson #2: ESB has bad exception management capabilities. 

Pushing the transformation towards the end-points effectively couples the end-points because these endpoints must share the format (*) of the messages they exchange and, just as importantly, they need to share data. Sharing data is usually done by one party having a procedure to export its data and the other parties using this procedure for creating local replicas of that data and synch-ing them according to a calendar. In addition to that the parties replicating the data have procedure for overriding the data they receive and adding to it. As far as I have seen it works pretty well and when it doesn\t work it is more of a management issue on the side issuing the data.

To sum this up, I would keep any data enriching out of the ESB and use the ESB trasformations only for format transformations and version transformations (**). One more benefit to keeping the ESB down to routing and format transformations would be that its role is well-contained and it doesn’t turn into a wild-card (does this particular problem gets handled by the ESB, how, why, when, etc…) making your IT assets more manageable. This results in a neater division of concerns in your overall enterprise architecture.

* It is advised that the format of these messages is not using APIs but rather text-based protocols such as XML, pipe-delimited, etc… API couplings is one of the most horrendous things that I have seen, synch-ing libraries on multiple endpoints at the same time is an incredibly time-consuming activity.

** Version-based transformation are transformations of the same message between 2 different versions of it: let’s say that one message producer decides to publish a new version of the messages that it has been produced so far, the ESB should be made aware of this and deliver the new version to the consumers that are registered to receive it as well as create a new message with the old version out of the original message and send this new message (formatted according to the old version) to the clients which are still registered to receive the old version. Part of this versioning effort is the transformation from the current version to an older version and this transformation would be best done by the message producer which will load a transformer into the ESB for this transformation. The ESB will then be re-configured to apply this transformation to the original message with the new version and the resulting message to be routed on its old route. This way the versioning effort will be pushed inside the ESB and messages will be delivered transparently to the intended receipients and recipients will not have to be upgraded when their counterparties upgrade their message format effectively de-coupling their release cycles.

However, one other reason for using inheritance that I see is for creating substitutes in order to partition code bases. In this case you are using sub-classes not because you need some shared behavior to be encapsulated in a super-class, but because you have distributed teams working on the same codebase and you need to minimize collisions between them in the codebase (a collision would be 2 or more persons working on the same file and is usually resolved by a merge in CVS). Solving conflicts thru CVS merge is a pretty error-prone process and doesn`t scale out well, so at one point it is necessary to partition development so that different teams can work on the same project without elbowing each other.
Sub-classing in this case would solve this problem by letting a developer substitute its team’s type for another team’s type.

One example is using helper classes which service a POJO and which are used by more than one team:
Let’s say that we have a TaxProcessor class that uses a TaxCalculator for calculating the taxes to be applied to a trade. Let’s say that you have 2 teams, one working on taxing bond trades and one working on taxing equity trades and that they are all using the same TaxCalculator class on the TaxProcessor. TaxCalculator has the method taxTrade which delegates to taxBondTrade and taxEquityTrade. As time goes on a lot of fixes are put into taxTrade in order to deal with various corner-cases to the point where 1) TaxCalculator becomes hard to maintain and 2) more than one developer is working on the same method at the same time. A solution would be to sub-class TaxCalculator into BondTaxCalculator and EquityTaxCalculator, have each implement the method taxTrade according to its specifics and then have TaxProcessor use the appropriate class according to the trade type.

As you can see from above good software models imply good separation of concerns. Good separation of concerns translates in good separation between development teams working in parallel on the same project. Good separation between development teams results in low transaction costs or interaction costs between teams and into a more efficient way to delegate work and in a development environment which scales better.

Yeah, like I would ever want to fork-over my content to Yahoo so that they can put an ad next to it. There is a sense of ownership to the content that you are creating (not necessarily monetizable) that would enrage many content producers if Yahoo would neglect it.

If search engines master this transition, how will the Search Engine Optimization crowd tweak their Web pages?

If search engines get around to this SEO will transform itself into a business that would protect your content from being pulled out of your site and plastered on Yahoo’s link-less search.

Firefox is a pretty decent browser, or more likely, a pretty decent browsing platform onto which you can install various plugins which will make you browsing experience better. Not too bad, even though I have a lot of problems with their address bar.

I think that what sets Google’s brower apart from the Firefox and IE, the browsers currently with the biggest market share, is the vast mass of data that Google has upon browser users and the way they will leverage this data to gain market share and the number of web-applications that it has in its portfolio. This browser will probably be integrated with Google’s other offerings such as GMail, Google Apps, etc… I think this sets them so far apart that the browser to come out of Google may not look like a browser anymore, but rather as an integration tool between your email (GMail), your RSS reader (Google Reader), your office documents (Google Documents), instant messaging (Google talk), etc… Who knows, maybe it will integrate even with Google Android ;-).

As I have said before, Google has put together a large amount of data that helps it describe the typical user that browses the web, and I think that this data will be leveraged in order to get ahead of Firefox because it Google may develop better ergonomics with of this data. Google will probably battle against Firefox for market share and this battle will be a battle between application designers (Firefox) and statistics on usage (Google). It will be interesting to see how it unfolds.

Later edit: I have been using Chrome for an hour or so and it feels more like a desktop than a browser. It manages to hide away, yet make available when needed, a lot of browser-specific functionalities. It feels like Firefox when run in full-screen mode, it gives you more space and manages to stay in the back-ground. A web-application should take advantage of all this space. The web is the new desktop…

My assumptions about Chrome’s design were largely wrong, from what I see Chrome is pretty much concerned with getting the browser out of the browsing experience by using a minimalist design rather than with implementing some funky ergonomics.
I think that its biggest hit is the extra space given to its user and I get the impression that  this extra space will make more of a difference for people WORKING thru the browser rather than simply consuming some passivly thru it, because this extra space makes you more productive. I don’t even see Chrome, yet with a few keystrokes I get any browser-specific functionality I need. I get the feeling that web-applications will need to be re-designed to work in Chrome and take advantage of Chrome’s extra space, for example Wordpress works atrociously in Chrome. 

Oh yeah, and the address bar is way better than Firefox’s. A few more tests and Firefox is history.

Later Edit: Firefox is history as far as I am concerned, I will be using Chrome.

Last thoughts on Chrome: Chrome is taking the browser out of web- browsing, it is almost invisible, when I am browsing I don’t see it at all, it has no edges and no menus, you only get the page that you are working on and its menus (this is why I said that web-applications may need to be re-designed in order to take advantage of the extra space that Chrome gives you, I am a bit confused when I am using Wordpress because where I now see Wordpress’s menus I was used to see Firefox’s menus). When I am toggling between pages I feel like I am toggling between 2 different applications.

Technical Debt is pretty much different from ordinary, everyday debt primarily because there is no creditor and no maturity date. Also, unlike regular debt, it is very hard to quantify. Technical Debt seems a bastardized version of regular debt in the sense that some development costs are identified as the principal (the re-factoring of the bad code) and some as the interest (the hacks used for dealing with this bad code) and it is built by taking some concepts out of the ordinary debt concept as the author sees appropriate in order to underline some development costs.
At the same time Tech Debt is defined exclusively from the point of view of the borrower who has to worry about paying the principal and the interest leaving out other actors (even though once you take the creditor out of the picture you have to wonder why you have to pay this debt at all ;-)). But, like Martin Fowler said at the beginning of his article, this is a metaphor, which leaves a lot of room for stretching various concepts. Fair enough.

However, Technical Debt seems to have been taken up and spawned quite a few siblings as you can see in this infoq article: liquid assets, moral hazard, fertile assets (???) etc…, which are either unfortunate enough to bear little or no resemblance to the original economic concepts they try to refer to or unfortunate enough to not refer to anything (fertile assets stands out as a prime example).
Let’s pick for example “Liquid Asset” which is defined this way: Perhaps the term “technical debt” focuses us on the wrong things; maybe focusing on the converse, on the investment side of things might be more effective.
First of all, a real-world definition: a liquid asset is an asset for which transaction costs are lower and for which there exists a market in which this asset can be sold and bought for a reasonable fee. Try to look into the above definition (or in this link which is provided next to it in the infoq example) and if you can spot the asset that is referred to, as well as the market in which this asset is sold and bought and the fees for these transactions please let me know.
Second, switching terms in order to motivate people displays how shallow these concepts are to begin with: if Technical Debt would be a concept that relates to somethings tangible in the real world, if it would manage to encapsulate a real problem or some real costs then Tech Debt should should not be swept under the rug for fear of under-mining the developers’ morale, but rather dealt with up-front because in doing so you would solve a real problem. The fact that Tech Debt is used the way this way points to the fact that it is so vaguely defined that few people can extract some somthing valuable out of it and that it can be stretched in any direction you want to.

What I see when I am reading the infoq article mentioned above are the first steps in the attempt to “marry” 2 fields which are pretty different one from another: economics and writing code and the excesses which appear when interest start to pick up in some vague concept (excesses similar to the introduction of the SOA concept). There is a desire to bring IT development under control and one way to look at it is to define costs and benefits associated with various actions and then apply economics to these costs and benefits in order to write code more efficiently. I don’t know where this will lead because we are at the beginning of exploring this.

As far as I am concerned I have developed an interest in economics a while ago and I read on it as much as I can. I also tried to “marry” the domains of writing code with and of economics, I even have an Econo-computing category, but I find it pretty hard. So far the only economic concept I found that can be applied pretty well to software development are transaction costs because they actually encapsulate pretty well some very real costs which arise when various entities are interacting one with another.

I will be following this Econo-computing field with a lot of interest and, who knows, maybe the concepts in this field will actually start to relate to something tangigle in the real world and will improve developer productivity.

I was watching this interview on infoq and I found that Gregor Hohpe pretty much voiced the same opinions, that as you scale out it is impossible to achieve state alignment at the time the operation and that you need to compensate for it by passing messages.These complexity costs are essentially the costs of managing business logic as it spawns across multiple systems, applications, departments, etc… State alignment in these cases consists primarily of having various business analysts agree on how these systems should interact (should this trade go to this depot for that security for that client) and then implementing the interaction between these systems. One thing that is not very obvious is that the cost of interaction between systems is pretty close to the cost of interaction between the departments running these systems and that as you scale out you need to be able to accomodate or change various departments you are encountering (*).

It is interesting to watch the interview from this point of view because you get to see the costly hacks that various organizations got to make in order to get their systems to talk to one another (the horrific “magic account” used in one bank makes one hair stand up). Gregor goes as far as saying that these ways to deal with these problems are pretty common, so common actually that at one point he gets defensive about the sloppiness which comes thru these examples. The fact that these problems are wide-spread is a pretty bad excuse, ideally these systems would be architected so that we do not need to add one layer of hacks on top of another in order to get something of value out of them.

At the end of this interview I ended up with the impression that we are on the first iteration of relaxing the ACID properties and that I got a pretty good glimpse into the costs associated with this relaxation. Sometime in the future patterns for relaxing these properties will appear and problems common in all these compensations will be solved. I think that these patterns will be more oriented towards business-logic and management rather than towards technology. Efficient interactions revolve around small transaction costs and minimizing transaction costs will be part of the solution of managing this growing complexity.

* Interestingly enough an SOA strategy will probably take these costs down if it manages to impose procedures so that various departments interact one with another more efficiently. However, I am pretty skeptical that we will see a lot of progress soon on this front.

Quickly I realized that I was cut-off from any source of information that I use because I access sites mostly thru their RSS feeds and my RSS feed reader was down. It was out of the question to go to those sites and read the news from their homepage (*) because those sites (BW, FT, Forbes,, Le Monde, infoq, etc…) have tons of content and I would not be able to navigate it in order to find what I like (this was what my RSS feeds were doing).
It was a pretty weird feeling to see how underlined how dependent I am on my feed reader and that my feed reader has become a single point of failure in my contact with various content providers. Take bloglines out and I am pretty much out of contact with pretty much anything I am following. And I don’t think that I can do much about it (**).
I can only hope that they come back up quickly…

* BTW, for a pretty interesting analysis oh how RSS changes browsing patterns please check out this post on Guy Kawasaki’s blog. The concept that “any page is a homepage” due to the various channels which are sending users to that page will change a few things in the content publishing business.

** Actually I could turn to my Google reader account to read the feeds from there, but this would mean that I need to update the OPML of my Google account, determine what stories are new and which ones I have read, etc… I could get past this single point of failure, but there are costs associated with it. Strangely enough, using Google reader as a replacement for Bloglines is similar to some work-arounds used for avoiding single points of failures in enterprise systems (synchronizing redundant machines).

Not a bad idea, the data would reside in this clustered cache and it would be made available to the application. One other thing that was considered was turning the whole architecture on its head and turn from the work-flow application into an EDA application. This is how it was supposed to come: the clustered cache came along with the capability of adding events to data as data was handled (inserts, updates, deletes, etc…). This would mean that while you previously had to have an MDB that was listening for a stock-quote request you could now code a stock quote object which you would throw in this clustered cache and which could listen for events. The clustered cache would have both flushed the stock quote objects to a database and would have call events on it. The stock quote object would in turn update other objects (let`s say positions for that stock) and while these positions would get updated they would fire more events in turn. A part of the business logic would be stored in these events and in the message passing that occurs between them, in a sort of event-driven-architecture.

The more I think about this, the more I get the impression that these events are nothing more than triggers in a database (the clustered cache taking on the role of the database). And coding your business logic in triggers is not a good thing if you listen to people with experience in database programming. Relationships between various entities and the interactions between them should probably be modeled at a higher level than at object level, when you are coding business logic in triggers you are essentially limited to that object’s scope.

Now, the idea of putting the business logic into these triggers comes from a pretty hard problem: the fact that for most artchitectures out there the data store is physically away from the code that is implementing the business logic, the database which holds the data and which needs to be consulted by the business-logic tier for data. Bridging this physical distance and the performance problems that come with it was solved in various ways: pushing the business logic into the database in the form of stored procedure, addind various caching mechanisms so that data is pushed towards the middle-tier, etc… In the example above part of the reasons for pushing the business logic into these events was you would effectively push the business logic into the new data-tier (the clustered cache in this case which would trigger these events). I would say that this is a pretty interesting approach with some benefits, but attention should be paid to minimizing the number of events (triggers) which get implemented.
A decade of PL-SQL development would recommend this…

I was pretty happy with Firefox 2 and I started using Firefox 3 only after a friend recommended it. I had some doubts, mainly when I was seeing the horrific address bar that Firefox 3 decided fit to implement. I have installed it, and after 3 days I wanted back to Firefox 2. My only beef (but a big one) with Firefox 3 is the address bar and the way it searches thru your browsing history and displays possible addresses you are trying to type. It is a great exercise in obfuscation, the new address bar manages to both hide away what you are typing (the font of the address bar is smaller than the font of the browsing history search results) and obfuscate the results of the browsing history search (bold-ing the hits of the word you type in the address bar causes further confusion as you are reading the results of the browsing history).

To make things even more confusing FF3 changed the way it searches the browsing history. Previously it was matching the address that you are typing against former URLs, now it tries to match the address that you are typing against URL, page titles, URL snippets, etc… in a pathetic attempt at mining your browsing history. In FF2 I used to be able to find my address with a few key-strokes, now my eye keeps scanning this list that FF3 dumps on the unsuspecting user:

I thought pretty seriously to go back to Firefox 2, but Firefox 2 will be discontinued in December. So I am stuck with this browser, let’s call it this way for lack of a better name. Frankly, I am waiting to run IE7 on my computer, it cannot be dumber than FF3.

Firefox 3 is to Firefox 2 what Vista is to XP. Period.

Later Edit Maybe I got too carried away, but what they have done to the address bar and to the address completion algorithm it is horrific. I really liked the way FF2 was displaying the addresses, it was clean, uncluttered and to the point. None like FF3’s address bar…

One reason that I see for this problem is the current misalignment between early education providers (such as high schools) and labor market. High school’s main goal is creating “better” human beings, its mandate is more social than economic. However with the tilting of the labor market towards jobs which require education it appears that education providers should also find a way to respond to economic concerns about the students entering an educational establishment.

I would say that we need shift the way we think about education towards economic concerns and insert some incentives into the educational process that reflect these economic concerns.

I don’t have time to finish this post, I will probably re-visit it later.

One very interesting thing that I come across pretty often is the fact that in large scale computing data tends to get de-normalized, basically it grows so large that you cannot fit the entities and the relationships between entities in the same database (this infoQ article is pretty good).
Let me give an example: suppose that you have table of users and a table or messages between users having these columns:
users: userID int, firstName String, lastName String, email String
user_messages: fromUserID int, toUserID int, messageTest String.
In the initial design a message from user A to user B would have only one record in user_messages and each user will be able to get the messages it has sent and the messages it has received from this table.
Now let’s say that the number of users sky-rockets to the point where you need to partition your database horizontally into a series of identical replicas. The problem that you will face now is where to store the data that is shared between these replicas, namely the table user_messages. You cannot store it either in the replica hosting user A’s data, neither in the replica hosting user B’s data and neither in its own instance because it will grow too large and because you will need to carry out a join over 2 physically remote databases. The solution is to drop user_messages for received_user_messages (from_user_name, from_user_email, message_text) and sent_user_messages (to_user_name, to_user_email, message_text) which will store the messages that a user has sent and the ones that it had received. Each database replica will have its own instance of these 2 tables each mapped to the user table. As you can see we are not storing the userID in the message tables, but rather the user information which was previously retrieved via joins.
Sending a message from user A to user B in this environment typically involves updating 2 tables in 2 separate database instances which usually is treated as a distributed transaction. However, you cannot carry out this distributed transaction because it is very costly so sometimes you resort to updating one table and sending a message to update the second table asynchronously (the overhead for delivering a JMS message is way less than the overhead for locking rows in the second database). In the case of updating the second table asynchronously you are still enforcing the relationships between the user table and the user message tables, but at a later point in time.
De-normalized data comes with synchronization costs: when you are updating the user information on the users table you will need to propagate the changes to received_user_messages and sent_user_messages so that the data stored in these tables will be up-to-date. This could be done via asynchronous processing as well, sending a message about data being changed on an ESB and having the concerned parties listening for and processing it. Synchronization costs should be watched very carefully because they could spawn a very high number of messages. Ideally we would synchronize data which is updated rarely (such as user information) in order to keep the number of synchronization messages down. Carrying out synchronization procedures in batches could be a way to deal with a large number of synchronization messages with the side effect that it may increase the latency with which the synchronization takes place.

Large scale computing typically seems to resolve around asynchronous processing (because in transactions message passing is cheaper than database access) and de-normalized data across with the overhead implied by it. The relationships previously enforced by normalizing data are usually enforced by passing JMS messages which carry out the data changes asynchronously. This design is driven primarily by the large volume of data which cannot be serviced by only one database and by the costs of carrying out a distributed transaction spanning 2 databases.
This is one design that I see emerging for large scale computing: de-normalized data along with asynchronous processes which are enforcing the relationships between entities via message passing.

It remains to be seen if this type of grid architecture will prevail in the future. Working against it is the emergence of multi-core processors which would allow for scaling up cheaply as envisaged by Brian Goetz in this article. If chips with hundreds of multi-cores and with large amounts of memory become reality scaling out could end-up costing more than scaling up and all the above could become history pretty fast. Scaling out will continue to make sense in some environments with humongous amounts of data (think Amazon, Google, eBay, etc…) but for the current fastest growing segment in grid applications, typically applications processing moderately large amounts of data, it may make sense to scale up once chips with a large number of multi-cores become a reality.

To watch…

P.S. These shifts in data processing (moving to a grid-type of architecture in order to accommodate the increase in data and then back to a single-box architecture because of the appearance of chip with a large number of cores) makes a pretty good case for abstraction in order to lower the costs of transition from one architecture to another. Ideally the architecture and the environment in which an application runs should not affect the business logic of that application. One way to insulate your application’s business logic from these infrastructure issues is by abstraction.

A pretty nice picture of the future, but I don’t think anymore it will happen at a large scale because I think that data is the primary way that a company uses for binding an user to its services, an issue that I have briefly touched upon in this post. If the datastores get opened then the cost of user migration from one service to another goes down and with it the risk of seeing your users migrate to your competitors. The data a user creates while using a service, and not service’s functionality, is what makes a user continue to use that service because migrating to a similar service implies losing what the data that it has created.

One other road-block to exporting data and using it outside of the place where it originated is the way it will interact with the outside systems and the possible problem of standards: how will data from one service be packaged so that it can be consumed easily by other services? Exporting the data in some RSS format and mashing it up in a Yahoo Pipes fashion could provide a way around the thorny issue of standards.
Security is another issue that comes to mind when dealing with exporting data.

For some companies binding users to their services by locking their data down doesn’t apply and Netflix is a pretty good example: Netflix can open up their users data because their service is about renting hard-to-find movies and not about hoarding user’s data in one form or another (email services, RSS readers, on-line newspapers, etc… are all about storing users data and applying some sort of functionality to it).

So we will have to wait and see how this un-folds. Frankly I do not think that data will get freed any time soon, in a world in which user loyalty exists only in history books companies will try anything it takes to bind users to their services and this includes data hoarding.
Data will probably get shared (later edit: but it will not be free in the sense that we can cherry-pick who is using it and who is not) between the different services within a large corporation not across corporations or between various companies thru partnerships, but data will not be freed without constraints. Controlling the way a service’s data is made available to other services will bind a user to that service even more because giving up on a service will mean giving up on the data stored within that service as well as giving up on the integration between that service and its partner services (within or without the original service’s corporation).

Later edit: This is one example of how our data will be shared among various entities. The new portal as envisaged by Catherine Holahan is simply a way of storing user information and disseminating it into its partner sites. Interesting to watch…

I have looked a bit at this product, but for now I do not have the time to try it out, so I read up a bit on it and read some blog posts. The award for the most imbecile post written about S2AP goes pretty easily to Marc Fleury who manages to get it wrong from the title: SpringSource`s Application Server. To him S2AP is an application server which is coming too late to the market, the market being currently dominated by, you must have guessed it, JBoss. Pretty hilarious as well as also his musings about VCs. The point is that S2AP is not an application server because it doesn`t provide the services that an application server typically provides, it is just a platform for putting together an application.

Another interesting post comes from Peter Kriens about S2AP extending the OSGi container: probably Spring Source is leveraging the market share it enjoys in order to add some headers that it finds necessary for doing a better logical partition of an application. It is confusing to the people behid OSGi, and probably a bit unfair(**), but I guess it it part of the game.

The fact that you have more control over the way you partition an application into bundles will drive the specialization of components further. In the future you will probably buy specialized bundles which target a very speficic functionality and wire them up in the SpringSource Application Platform, Billy Newport`s hits the nail on the head in this post. You will not need to buy an application server from IBM or BEA or someone else only because you need a state of the art transaction manager and not use, while paying for, the rest of the capabilities that that application server comes with, but rather you will be able to buy only the transaction manager, plug it into your application as a bundle, expose its services and then integrate it into your application’s components.
This is probably where Spring has the potential to do the most damage to the traditional app servers vendors in the future because it will force them to un-bundle their app servers into offerings that will be sold on the market as single bundles (a transaction manager, a JMS engine, etc…) rather than integrated solutions. It remains to be seen how this will un-fold, but the potential for damage is there. The road towards un-bundling of application servers into specialized components is a very disruptive process (*) to both app server vendors and their customers and as a result it will take a while to become mainstream but I think that this is where we are going: to specialized components bundled into applications.

I would conclude by saying that the future of S2AP is a bit uncertain, it is a brand-new product targetting a fairly new market, the market for composing applications out of components in OSGi (***). If there is a need for this type of application composition probably Spring will occupy a large part of the market leveraging their brand, their product portfolio and know-how.
I wish them a lot of luck in this enterprise and I will follow it closely.

* I think that this is the most disruptive product to be released by the Spring people since the IoC container. It took a while for the IoC container to start having an effect in the JEE world, but once its benefits were recognized it revolutionized JEE. Let’s see if this will happen again.

** As Glyn has pointed out the OSGi-like headers are syntactic sugar. They created a bit of a confusion, but apparently it is getting straightened out.

*** Later Edit: I don’t know why but I get the impression that setting up a marketplace for these components would help their adoption. At the very least this would open a venue for people interested in trying them out and people interested in selling them.

Except that I find right now that I am pinned to Bloglines. Let me explain.
Any feed reader lets you export your OPML file which contains your feeds grouped into categorie and this feature pretty much lets you travel from feed reader to feed reader without many headaches. No feed reader would take its users hostage by refusing to export their OPML file because every user will refuse to be taken hostage in such a way.
However, if you use Bloglines’ pin feature a lot you will find yourself hostage to Bloglines because you cannot export the items that you have pinned and leaving Bloglines would also mean leaving behind the items that you have pinned and that are probably important to you. You cannot export these pinned items typically because this is a feature which is not widely-used by other feed readers and when it is used by other feed readers it is implemented in a variety of flavors which makes its export inefficient.

So I would say that I pretty much taken hostage by a feature. A niche feature, which makes a pretty interesting case for niche products: if you manage to get your user to create data while using your niche product, chances are that your user will be hostage to the data that it created and the feature consuming this data because this data cannot be exported easily to your competitors’ services. Pretty interesting…
P.S. Initially when I started pinning items on Bloglines I realized that I will be taken hostage to this feature and I thought about bookmarking them on del.icio.us. But I decided that I am too lazy for this. So I guess I’ll have to use Bloglines for quite a while…

I am wondering if the IOC realizes what these events are doing to its brand (*) because lately the Olympic torch has become the symbol of collaboration with a brutal regime which has to be put out rather than a symbol of peace and fraternity. I am also wondering how things will play out in Beijing, because this is where the next episode of this saga will probably play. Right now the torch is safely out of the Western world where dissent is allowed (**) and it is passing thru a string of authoritarian regimes where it will not encounter any serious resistance.
I would not be too surprised if the Olympic stadium will be empty, most of the Olympic tourists preferring to stay at home. I would not be too surprised if the Chinese government would fill this stadium with people picked up at random from the street. And I would not be too surprised if we will have some similar incidents in Beijing as well. If we do have these incidents the last place that I would be in is in IOC’s shoes: they would pretty much have to wait on the sidelines and pray that the Chinese government will not quell these dissenters (which happen to be all IOC customers) with too much brutality.

Personally, I think that these events are good. They are bringing home to the IOC the dangers of doing business with dictatorships: they could devalue your brand and beat the hell of the people that believe in your values in front of the cameras. Maybe they will think twice before awarding the place where the next Olympic games will take place…

* Unlike the IOC the Chinese government doesn’t have these PR problems. It will probably continue to act the way that it does, brutally that is, after all its interests lie exclusively in providing cheap labor to the Western world and using the gains from it for consolidating its regional position. China is light years away from having any meaningful dialogue with a regular guy living in the Western world.

** The fact that the IOC banks on the success of this torch-bearing missing on collaboration with authoritarian regimes will not be lost on many. If I were in a decision-making position within the IOC I would play down the passing of the torch thru these countries because it would stress this collaboration even more. After all, IOC’s main customer is living in the Western world and you probably don’t want to alienate it by playing down its values…

Knowing how a package implements some business requirement may make a developer pick up a method from that package and use it because it fits its requirements. Unchecked this may lead to spaghetti-code, pretty much everybody calling methods from all over the place in order to get their job done quickly.
Using restrictive access attribute (such as private methods) is not really a work-around, if someone thinks they need a private method to be made public because they “know” the package they will usually make it public without any second thoughts.

The right approach to interacting in a new way with a package or system is to delegate that interaction to that package, i.e. to code this interaction in the package itself if possible. As I was arguing in a previous post knowledge about a package is one of the transaction costs of interacting with that package. Transaction costs usually delimitate who does what because resources will typically cluster together into larger entities in order to bring these transaction costs down. Transaction costs, when handled correctly, will prevent methods whose usage requires extensive knowledge (and which carry large transaction costs) to be called from all over the application. Methods that get called from all over the application are typically methods about which little knowledge is required, helper methods being a good example.

So I would advise people to either make sure that whoever is using their packages either understands them well or very little, having people in the middle that think they “know” the package may result in incorrect usage of their packages.
I would also advise people to refrain from spreading knowledge about their packages indiscriminately and rather target the recipients of this knowledge carefully.

You have to wonder what is the relationship between Walmart and the producers of fair trade coffee: does Walmart reins in the urge to apply its enormous clout in order to drive the prices down on these producers, most of whom are pretty small and un-organized? Or it does apply its clout, but reluctantly? Or it doesn’t exert any price pressures on fair trade coffee producers and uses fair trade coffee to increase its appeal among a particular demographic?

Strange, I was a bit shocked to find this on a Walmart shelf…

BTW, the price for that coffee was pretty high by Walmart standards, I could have bought 2 1-liter coffee jugs (I can’t believe that coffee can be bought this way unless you have a coffee shop or something) for the price of 250 grams of fair trade coffee.

There are generally 2 sides to the privatization of health-care debate: the side pushing for the privatization of health care services, typically pro-business, and the side wanting the government to run health care services, typically leaning towards a social democracy-type model.
The pro-business camp’s opinion is that privatizing health care services would make them more efficient and would also transfer the burden to the ones willing and able to pay for them.
The pro-social-democracy camp argues that this would cause an exodus of health-care professionals from the public services, which are barely coping with the current demand and which pay less, to the private services which would pay more and where health-care professionals would work under a smaller work load due to the filtering of “customers” by price. They argue that this would result in further degradation of the public health-care services, due to an even smaller number of professionals working for it, which in turn would raise the incentives for leaving it. The public health-care services would enter a down-ward spiral whose end-result would be that the people using the public services, the neediest in this case, would have no access to health-care services anymore. I subscribe to the first part (the public services would degrade over time), but not the second (the down-ward spiral) because there are ways to stop this exodus.
For the health-care as a public service camp the smallest chink in the public access armor would essentially rip it open to the onslaught of private health-care.

The fact is that what both camps are focusing on is the way health-care services are distributed to people and not on the imbalances between the supply and demand of such services. Anyone with a pair of eyes would see that the demand of these services greatly exceeds the supply: long waiting times in emergency rooms, people parked in the hospital corridors for days, etc… And here things start to get interesting…
Let’s suppose that you decide to privatize health-care completely and let the market decide who is paying for health-care that it consumes. A good or a service that has the supply and demand imbalances that health-care has would see its price get big pretty fast. Big prices mean also big profits, and there is no surprise here that the pro-business lobby is pushing towards privatization because they stand to gain quite a lot from it.
Now let’s suppose that you decide to keep the status-quo and allow health-care to be delivered only thru government agencies. The result would be that everyone would get the same pathetic service or no service at all, and this is exactly what we are getting right now: everybody gets to wait 5 hours in an emergency room regardless of how wealthy he or she is.

Both the pro-business group and the social-democrats are arguing over the means to distribute this very small supply of health-care services to a population that has a high demand for it. Leaving health-care to the market would ensure that only the wealthy would be have full access to it, leaving it the way it is would ensure that we are all getting the same service. It is essentially an argument over the way to distribute this scarce resource and over the winners and losers that come out of each particular distribution mechanism. Each camp is trying to prove that its distribution mechanism will also allow for the supply to grow faster, but I am pretty skeptical about it (*). So I would argue that we should look at the root of the problem and try to stimulate the supply of health care professionals and to enlarge the med schools quotas to reflect the current demand for such services.

Except that we have a bit of a problem (**) if we simply try to : we are leaving next to the US which is more than happy to let the whole health-care field to market forces. As explained above the result is that in US (which experiences pretty much the same imbalances between supply and demand) the price for health-care is very, very high along with profits and salaries among other things. If we increase the supply in Quebec we run the risk of seeing health-care professionals leave for the US in search of better salaries and reduced work-loads (***), an exodus similar to the possible exodus from public health-care agencies to private health-care companies.

So we have a problem. This pretty much is known. What solutions to this problem? Frankly, I don’t know. I think that we need to increase the number of health-care professionals running the risk of seeing some of them leave for US. In US things could be changing if the democrats get the White House (some health-care would be administered thru government agencies driving down prices and salaries) and this could dampen the incentives for leaving to US.
Hopefully we will manage to get out of the grid-lock that we find ourselves in…

* The pro-market argument for growing the supply to align itself with the demand is the following: a good which experiences high demand but a low supply sees its prices rise along with the profits. Entrepreneurs will realize that it is profitable to join this market because of the high profits and will start selling more of this very profitable good. This would in turn will raise the supply to match the demand for it and inevitably bring down the prices.
I don’t buy this argument because in health-care the supply is pretty inelastic to demand since it takes a very long time to “produce” a doctor, about 15-20 years. In order to have the supply align itself with the demand you would have to carry out extensive demographic studies from which to determine the future demand for health-care services and tweak the med schools admission rates for the future demand. The fact that supply is inelastic to demand in health-care make say that the only thing that a market-based health-care system will achieve in Quebec is to redistribute resources and not to redistribute resources plus create new ones.
There are probably some services within the health-care industry in which the supply is more elastic to demand and it would be a good idea to leave them to the market.

** It is a problem or an opportunity depending on how you look at it: high prices in US could drive the government of Quebec to sell some health-care services to US and use the money which these services would generate for funding other health-care services.

*** The fact that we have the US in vicinity renders a lot of the comparisons between Quebec and France moot: a French doctor doesn’t have the option of packing up his or her belongings in a minivan and drive 300 kms to his or her new workplace in US. This puts a downward pressure on health professionals salaries in France thus reducing the health-care costs.

P.S. I am seriously thinking about starting to write this journal in French, but I don’t have much experience in writing in French and I think the first posts would be absolutely ludicrous (I will be probably missing all the accents ;-)). I’ll see what I’ll do.

P.S.S. I like economics and I wrote this post mainly for applying my knowledge of economics to what I know about health-care in Quebec. I have not researched deeply how health-care is being regulated and implemented in Quebec, I just wanted to try out some concepts. Hope I got them right…

Regarding my use of testing I would say that JUnit pretty much fills my bill and I don’t see a need to move to TestNG. I was watching the presentation not so much as to know more about TestNG, but to get some exposure to the market for testing products (*). So the features which were presented and which were said to be in a high demand from the users were the ability to define test groups, to test data connectivity and to define dependencies between tests (**). Pretty fair, I would take this functionality to be a logical progression from the ability to run a bunch of tests at random as JUnit lets you do it, you are basically starting to look at the tests you run from a higher level and you start looking for ways to aggregate these tests into higher-level constructs, such as test groups, which can then be manipulated according to various needs.
So far I would say that Test NG is looking at ways to manage the complexity coming out of a high number of tests. Test NG is for test-heavy shops, where testing is considered a concern on par or close to development. Not a bad thing and I pretty sure that TestNG covers some functionality which is in high demand by Java developers.

There is, however, one need for testing that so far it is largely unsatisfied and sorely missed: the need to test workflows or series of events. Let’s say that you have an application that is a series of MDBs, each accepting messages, transforming them and then outputting them to the next MDBs. You would want to be able to test this application end-to-end.
Let’s say that you have an application that receives market trades, needs to process them and then transform them in order to send them downstream to tax systems, settlement systems, etc… You would want to put thru a trade, set-up in a certain way and then trace its execution thru these flows and determine if, in what stage and in what shape has this trade reached the Application-to-Tax-System gateway.
When I think about work-flow testing I usually think about defining a message, inputting this message into a work-flow engine and then defining interceptors to see how the original message has been manipulated at various stages. You may need to test both the work-flow itself (to see if the message has been going thru the work-flow that it needed to go thru), the end-result (to see where the message has been forwarded to and in what shape) as well as how it behaves at various stages in this workflow (if needed). I see work-flow testing primarily concerned with interception of messages (and this would probably be a great use of AOP) and with the possibility to correlate messages passing thru this work-flow with the original message.

OS testing tools so far are limited to synchronous testing. I wonder at what point will the need for work-flow testing become so pressing and the demand for it so great so that one OS shop will start doing something about implementing a testing framework for testing workflows. Then we could use Event-Driven-Testing for testing an application written in a Event-Driven-Architecture manner…

* I know that trying to form an opinion on a market such as the market for OS testing tools from a vendor presentation is a pretty risky business, what you get from vendor presentations is usually distorted because of bias and time constraints but I will assume that this presentation gave a fair image of what users want from testing tools.

** From what I know TestNG is a lot more than some annotations that give you the possibility to form test-groups and test dependencies. However, I would say that these issues are probably considered more important and more aligned with the market for OS testing tools since they were the ones which were included in this presentation.

P.S. If I were to choose one enterprise concern which is not addressed by the Spring stack I would choose work-flow. Spring has modules for integration, batch, transaction management, security management, connectivity to various end-points, etc…, it is missing this capability. From what I see work-flows are used pretty heavily in the enterprise space and for now I think they are mostly implemented either by in-house solutions or by commercial solutions. OS could probably make a contribution in this space as well. And if it does it should probably try to give the developer the ability to test work-flows, or at least make it easy for him/her.

P.P.S. The comments by Alan Keefer on this thread reinforce my beliefs that we need to carry out tests at every level, from high-level to unit test. For a work-flow based system this would mean that we need to test the work-flows themselves and not only the units making them up.

Later edit:  There are actually a few OS workflow tools. OSWorkflow, the ones at the bottom of this article and probably a few others. I should try them at one point.

This is a very interesting take on compensations and his proposition (that compensations are regular business activities that are part of a business process and that could be coded as such) gives a lot of flexibility to handling compensations. However, is exactly this flexibility that will probably force someone away from the compensation scheme devised by Bill and back to WS-BA. As Mark pointed out you still need to enforce end-points to run the same version of the communication protocol, be it WS-BA or an ad-hoc REST-based protocol.
However, I would go further and say that the flexibility devised by Bill runs counter to market acceptance. Standards are straight jackets which people choose to wear when it makes sense, typically when it makes some process (such as interacting with a different system) more efficient. Coding a business activity in WS-BA would enable you to plug it effortlessly different systems implementing the WS-BA standard, in theory at least. You would expose your application to the outside world effortlessly and you could interact with a greater number of systems because you have chosen to trim down your application to fit into the WS-BA straight-jacket.

On the other side, if your application cannot be forced into the WS-BA standard probably it would make sense to drop this standard and expose the compensation logic as you and your partners agree. It would not be the first time that a standard gets ditched for a proprietary solution particular to a few partners and certainly not the last time. Sometimes a straight jacket is just too stifling…

I was making my way along the streets of downtown Montreal cursing the weather like everyone around me when I see this guy waiting at a red light. He’s dressed in a suit and a tie, in sharp contrast to the sea of parkas around, with his face and knuckles bluish red from the cold. The shoes, not winter boots,  are ankle-deep in the chipped ice that is decorating our streets right now. He is fixating the red light waiting for a chance to keep going, who knows where, while fighting the wind that is trying to unbutton his suit. I kept wondering how did this guy got parachuted into this environment and where the hell he left his winter-coat. I also wondered at how this guy was determined to keep going, oblivious to the horrific conditions around him.

A fine picture of determination…

However, the article shone light on the fact that you cannot ignore the team when embarking on a new project and tried to explore 2 different problems: the effects of team distribution and of the team skills on the architecture of the project.

Let’s start with the team skills: the article argues that when you embark on a new project you should center its architecture around the strengths of the team. Well, this is so obvious (you need to utilize the most of your team) that it doesn’t need to be re-stated. The problem that he posed is a bit unusual: the architect thinks that Ruby should be used for a project, however the team only groks Java. He goes on to say that the architect should probably use Java and try to evangelize Ruby in order for the team to learn Ruby. Well, the team will probably know some Ruby 6 to 12 months into the evangelization process, at which time a lot of development has been done in Java meaning that it is too late to contemplate using Ruby on this project. Maybe next time…

The other subject of the post was how team distribution affects architecture and the author of the post shows a bad way of architecting an application without any regard to the team distribution and composition. I would say that he is right about it, when working in distributed team it is essential to be able to implement a certain amount of autonomy within the teams so that each team can make decisions on its own and not get grid-locked on dependencies from other teams. However, setting up autonomous teams is mostly a managerial issue and not a “pure” architectural issue, as you can see the borders between architecture and management start to blur in certain cases.

It is a pretty interesting article which outlines some recent developments in today’s IT environment: polyglot development and distributed teams. These issues are managerial issues (maximizing resource utilization) which are cloaked in architectural clothes (if I can use this expression ;-)).

However, I started realizing that maybe DTOs should not contain any complex behavior and that they should be as dumb as dirt, especially when being used for communication between multiple systems. Consider this example: we have a tax system which uses the object TaxInformation which various systems use for getting information about taxes. Let`s say that the commodity trades system, the equity trades system and the FX trades system are all using it. Let`s say that this object looks like this:

public class TaxInformation{

private Trade trade;

private TaxReceipt taxReceipt;

public TaxInformation(Trade trade){

this.trade = trade;

this.taxReceipt = new TaxReceipt();

}

public TaxReceipt getTaxReceipt(){

return taxReceipt;

}

}

As you can see a DTO dumber than dirt. Let`s create an endpoint creating and serving such an object:

public class TaxService{

public TaxInformation getTaxInformation(Trade trade){

TaxInformation taxInfo = new TaxInformation(trade);

return taxInfo;

}

}

All is fine, but a requirement comes in which says that if the trade is restricted then the tax receipt of the tax information object related to that trade should also be restricted. Here you are presented with a few choices:

1) Implement this requirement in the TaxInformation object probably in the constructor.

2) Implement this requirement in the TaxService object.

3) Implement this requirement in the TaxReceipt object, in its constructor. For the sake of argument, let`s assume that we do not have this possibility, so I turned it off.

OOP purists will immediatelly recommend choice #1, because this would turn the DTO into a rich bean and would do away with the anemic DTO. This is a mistake, because all the systems using this object for communicating with the tax system will start having class versioning errors unless they are updated with this DTO`s new class. The problem is versioning such a DTO. Putting complex behavior in a DTO raises the risks of change in the DTO, which raises the effort of propagating this change in the systems which use this DTO for communication. At one point it makes sense to drop DTOs and use some sort of protocol for communication, preferrably a protocol which deals with change pretty easily (XML documents with schemas which always get extended and with constraints which get changed very rarely are a pretty good fit).

DTOs should probably be used in systems whose lifecycles are kept in synch (i.e. they get upgraded in synch), in such systems you can propagate type changes pretty easily because you have a certain amount of control over their lifecycles (*). Expanding on this, I would say that typed languages are probably most effective locally because type changes cannot be propagated over large distances efficiently. Type-coupling (or API coupling) is a very hard coupling and its should be avoided and replaced with protocols. DTOs do create an anemic model and should be avoided if possible, but if you choose types for communication between systems it would make sense to keep these DTOs to their most basic function of passing data around or to have procedures for propagating type changes in all the systems using types for communication.
Sometimes weak DTOs are actually a good thing…

* BTW, whether a set of systems whose lifecycles are kept in synch is actually one big system with one particular lifecycle is a pretty good question. I would say that you can think about it as a big system.

Voyage, voyage
Plus loin que la nuit et le jour, (voyage voyage)
Voyage (voyage)
Dans l’espace inouï de l’amour.
Voyage, voyage
Sur l’eau sacrée d’un fleuve indien, (voyage voyage)
Voyage (voyage)
Et jamais ne revient.

Sur le Gange ou l’Amazone,
Chez les blacks, chez les sikhs, chez les jaunes,
Voyage, voyage
Dans tout le royaume.
Sur les dunes du Sahara,
Des iles Fidji au Fujiyama,
Voyage, voyage,
Ne t’arrêtes pas.
Au dessus des barbelés,
Des coeurs bombardés
Regardent l’océan.

Voyage, voyage
Plus loin que la nuit et le jour, (voyage voyage)
Voyage (voyage)
Dans l’espace inouï de l’amour.
Voyage, voyage
Sur l’eau sacrée d’un fleuve indien, (voyage voyage)
Voyage (voyage)
Et jamais ne revient.

Au dessus des capitales,
Des idées fatales
Regardent l’océan.

Voyage, voyage
Plus loin que la nuit et le jour, (voyage voyage)
Voyage (voyage)
Dans l’espace inouï de l’amour.
Voyage, voyage
Sur l’eau sacrée d’un fleuve indien, (voyage voyage)
Voyage (voyage)
Et jamais ne revient.

The post looks for similarities between implementing some piece of functionality in an asynchronous manner and implementing the same piece of functionality with methods that do not return values. It pretty much finds these similarities, but there are some important differences which we will explore now.
Let’s take a look at some code: suppose the you have this purchase order component which has the method placeOrder for placing an order:
public PurchaseOrderBean{

public placeOrder(PurchaseOrder po) throws PurchaseOrderException{

try{

// update its status in the DB

changePurchaseOrderStatus(po);

// execute payment

paymentBean.performPayment(po);

// update the inventory

inventoryBean.updateInventory(po);

// integrate with the back office

backOfficeBean.notifyBackOffice(po);

}catch(BackOfficeException ex){

// handle BackOfficeException, possibly rolling-back the current transaction

}catch(InventoryException ex){

// handle InventoryException, possibly rolling-back the current transaction

}catch(PaymentException ex){

// handle PaymentException, possibly rolling-back the current transaction

}

}

}

As you can see in this case the method placeOrder simply tells the inventory bean, the payment bean and the back-office bean to do their work.

Now let’s implement the same method in an asynchronous manner using JMS and QueueSenders:
public PurchaseOrderBean{

public placeOrder(PurchaseOrder po) throws PurchaseOrderException{

try{

// update its status in the DB

changePurchaseOrderStatus(po);

// execute payment

queueSenderToPaymentJMSQueue.send(formatMessage(po));

// update the inventory

queueSenderToInventoryJMSQueue.send(formatMessage(po));

// integrate with the back office

queueSenderToBackOfficeJMSQueue.send(formatMessage(po));

}catch(JMS exception){

// handle JMS Exception. Usually it means rolling-back the transaction and sending this exception up-stream.

}

}

}

As you can see the biggest difference between the 2 implementations is exception handling. In both cases you are telling a component (in the first case a bean, in the second case a different system) to carry out its work but in the first case you get to deal with the exceptions. One big difference between asynchronous processing and synchronous processing that I see is that in the case of asynchronous processing you are delegating the whole implementation of a particular functionality including failures while in synchronous processing you can delegate dealing with failure to the method that orchestrates the interactions between these components.
I consider this difference to be important when designing systems because it essentially splits tasks and responsabilities.
The post was right on the money on the need to try to avoid methods returning values which need to be dealt with in the calling methods, because coding in such a way encourages to write components to which you can delegate a task better. Having to deal with methods that return values means basically that you need to interpret these values outside of this method and this means that not everything was delegated to the method. Sometimes you need to code this way though…

Now let’s dig a bit into the way exceptions are handled in this 2 pieces of code. Let’s suppose that the payment fails for this particular purchase order, the credit card is expired. In the first example all processing would basically stop at this step, we would roll-back the transaction and notify this error upstream.
In the second example it is a bit more complicated, because the payment would fail in the payment system which would then have to notify the back-office, the inventory system that that particular purchase order has failed (probably the most efficient way to do this would be to publish a message about a failed payment on an ESB channel to which the other 2 systems are listening. Connecting the payment system directly to the systems it is related to is a tight-coupling and should be avoided. *).
Exception handling is done differently in the 2 implementations: the first implementation it deals with the exception right at the source while the second implements exception handling by passing messages between the systems in the back-ground. The second implementation is also more prone to tight-coupling between systems.

One cost involved in the above implementations is the cost of exception handling . In synchronous processing exception handling is done upfront and it is usually not expensive since you are handling it at the source while in asynchronous processing exception handling usually ocurs behind the scenes with systems passing each other messages, you should pay attention to it when designing systems. If you could delegate the whole processing, including failure handling to a component it would probably make sense to use asynchronous processing. If you find that some failures imply multiple systems it would probably make sense to use synchronous processing.

Another cost involved in the above example is the cost of locking down resources: in the synchronous example this cost is pretty high: you need to start a transaction, start processing, lock down database rows, etc… and roll back that transaction in the event of failure while in the second example this cost is extremely low: you start a transactions, you send some messages and you commit the transactions.
You essentially have these 2 costs and each implementation has a different cost: the asynchronous implementation has a high complexity cost (passing messages about exceptions behind the scenes) while the synchronous implementation has a high physical cost (the cost of locking down resources pending a transaction commit). These cost structure should be kept in mind when designing systems (**).

I would end this post saying that while most of the people seem to associate asynchronous processing with low-latency method calls, delayed execution and relaxed time constraints it is important to bear in mind that asynchronous processing should also be associated with the way you delegate a particular task to a particular system which is primarily a management issue. What I wanted to stress is that asynchronous messaging forces you to fully delegate processing (including exception handling) to a different component or system. It is this hand-off that I see as just as important as the other things that asynchronous is associated with: faster return from the call because you are simply passing a message rather than waiting for its execution,etc….

* One caveat though. It is a good thing to try to keep the number of the message types that are sent thru an ESB under control, otherwise you will end-up with a Spaghetti Oriented Architecture ;-).

** The present push away from ACID transactions towards systems passing messages between each other would imply that the physical cost is larger than the complexity cost. At the same time ESBs are becoming more and more popular in order to keep the cost of the relationships between various systems low.
Transactions can be viewed as a way to enforce state synchronization between multiple actors: these actors try to carry out a series of actions at the same time and if one of them fails they should all go back to original state. ACID transactions are enforcing state synchronization at the time the actions are carried out, however in some cases you can carry out this state synchronization later.
Please check these links for some unusual transaction mechanisms. Some of them (especially XTP) will very likely become de-facto standards:
Are Cross-Service Transactions A Violation of the Autonomous Tenet of Service Orientation?
Xtreme Transaction Processing under SBA terminology
Large-scale distributed transactions
Constrained Tree Schemas (CTS) and applications (CTA) for extreme OLTP (XTP)

Note: This post was written while listening to Mylène Farmer live at Bercy 2006.

P.S. This post is also quite all over the place: exception handling in async interactions vs. synchronous processing, transactions, etc… I guess Mylène Farmer can be quite distracting ;-).

You could speculate that Google is not lifting the curtain on parts of its infrastructure for fear that it would undermine one of its competitive advantage: the ability so scale out almost effortlessly. You could speculate that the large traffic sites which use grids for strategic reasons will be reluctant to open up their architectures to the outside world to the degree that Amazon has done and to a certain extent I see their absence from this space as a the cost of keeping inner workings of your organization away from your competitors eyes. Amazon probably has gone thru the same stage where its proprietary architecture was one important differentiator between itself and its competitors, but it looks like it has gone beyond it, it has grown beyond . Probably as Google, eBay, Yahoo, Microsoft will mature they will also start opening parts of their infrastructure which are not creating a difference between their services.

But I still don’t understand why Sun’s is silent in this space. Part of its strategy to move towards open source would have been to move towards services at the same time (since your code doesn’t pay you need to make money from running services on this code). Support is one type of service, but I see grid computing as one service which is far more profitable to Sun.
Cloud computing comes with a pretty high entry barrier: you need know-how, operation teams, investments in hardware, etc… in order to run the large data-centers that are hosting this service. Sun could have tried to do that and in the article at the beginning of the post they seemed pretty determined to open up such an environment. Judging by the size of their investment in MySql you could say that Sun it committed to software. I don’t see hardware playing a big role in the future, but rather infrastructure. Once you solve the problems of scaling out the need for big iron boxes diminishes, however the infrastructure starts to matter a lot. The only thing that Sun seems to do is to sell various grid-solutions but not to host grids, I think they are missing out on a big piece of what will be demanded in the future IT landscape.

So how do I see Amazon’s grids doing in the future? I see them doing pretty well, for now the market for grid computing on a credit card (to quote Cameron Purdy) has only one seller, that is Amazon. Amazon will bank on the integration between small companies and its grid capabilities, integration which translates in in the number of VM images stored on EC2, on the different configurations that different hosts are using, etc… This integration is essentially an infrastructure lock-in and once Amazon has acquired these customers it is a safe bet that they will not migrate to its competitors (which are yet to appear) easily.

Later edit: I was thinking about using Amazon`s EC2 to host Facebook applications which start having operational problems because of becoming immensely popular (as illustrated on this post on Marc Andressen`s blog). It would probably make sense, but I didn`t dwell too much on it.

Design skills are largely the outcome of some soft skills (good communication with the customer, easy understanding of a domain, maybe knowledge of a few domains, etc…) that are much needed by any developer. When you bring design skills in-house you also bring those soft skills in. These soft skills will translate in a more frictionless environment, better communication with the client, faster ramp-up on new projects for which the person with design skills has domain knowledge, etc…

Hiring a person can be viewed as an investment: you are investing the salary for the probation period, the relationships that this person will make (these relationships may included relationships with your customer).

Hiring a person comes with opportunity costs: you will probably have to refuse the same position to some other one.The opportunity costs of a hire are the skills of the person(s) which has/have been refused the position and the contribution that the refuse would make in the future, it is important to keep these costs down as well.

The opportunity costs of choosing a person with design skills vs one with language skills is largely the time it takes the design-skills person (I am getting tired of typing this denominator) to learn the language. I would trade this opportunity cost vs. the opportunity cost of hiring someone with no design skills because I feel so (sorry, I don’t have time to go into details).

Reading the above you will see a huge bias towards design skills vs. language skills in this post to the point of where you are probably wondering when should you hire a person with strong language skills (or who knows, maybe if you should hire such persons at all).
Staying in the same controlled thought-experiment environment I would add that positive bias towards a particular set of people tends to rise those people’s salaries, they become more expensive as the demand for them increases faster than their supply. It would follow that at one point the premium associated with design skills would become so large as to dwarf the benefits from employing people with design skills. At that point it would become economically sensible to hire language-skills people.
Another point would be that you could lower the opportunity cost of refusing a person with design skills by hiring a person with language skills which is motivated to sharpen his/her design skills. If you come across such a person the opportunity cost of hiring his/her would be the contribution made by the person with design skills while the person with language skills is still in the ramp-up period of sharpening his/her design skills. I would say that this opportunity cost would be more than offset by the relationship that you will develop with the person with strong language skills while mentoring, tutoring his/her.

And this is the last time you will read design skills in this post, I feel like I said it 500 times ;-).

P.S. I guess it shows that I love economics ;-).

I hope it will not create much of a disruption.

Probably the number one problem of distributed teams working on the same project is to make sure that the code doesn’t contain references to methods which don’t exist or which have changed. Statically-typed languages solve this problem by their own definition at compile time. Dynamically-typed languages probably have some solutions to this problem (I heard some people documenting the Ruby code that they write), but the biggest drawback of them is that this type of solutions are human-powered and not automated, an automated solution would be something similar to a compiler inspecting your code for invalid method calls. I would say that the solving code-conflicts that arise in distributed teams are more expensive in dynamic languages than in static languages, therefore when choosing a dynamic language you should probably stay away from coding in a distributed development environment (*).

One problem that I see with Ruby and all those elegant languages is that they are not very expressive to a large audience. They encourage brevity, but I find that they encourage it at the expense of readability and this narrows their audience (this is why I find it unlikely that Ruby will break out of a small elite). A developer working on an application needs to understand pretty fast what a particular method does rather than marvel at the “beauty” of that particular piece of code. I find that the information stored in Ruby code is harder to extract than the same information stored in Java code. It is “beautiful” (**), but also inefficient and it is more expensive to work with it.

I would say that the biggest challenge using dynamic languages revolves around collaboration and communication. Statically-typed languages put a smaller cost on communicating thru code, dynamic languages can achieve the same thru some external mechanism. While dynamic languages have more features, this freedom comes at the expense of collaboration. Evidence would seem to make this supposition true, most development in dynamic languages happens in small teams concentrated locally where the needs for communication are small.

* All the above being said I remember Martin Fowler talking about agile projects coded by distributed teams which did very well. He didn’t mention what language these teams are using, but given the fact that ThoughtWorks seems to favor Ruby I would assume that these distributed teams were using Ruby as well. It would be interesting to see the steps taken by them in order to be able to handle these communication problems.

** I personally find it brain-dead to talk about the “beauty” of code because this is not the purpose for which code is written. A statue should be beautiful because this is its purpose, to be enjoyed by who watches it, a piece of code’s purpose is to work as it is supposed to.

Later Edit: The communication and collaboration costs of dynamic languages will define the population of developers that use them and their usage within that population. I would not be too surprised is most of Ruby and Groovy usage will be restricted to small scripts which do not require a lot of collaboration while a small niche of very productive teams will be able to use them on a large scale.

You have to wonder what will happen to Java in the future. Will it disappear and turn into a dinosaur that could not adapt to a changing IT environment or it will manage to survive the problems that it has at the time? One good look at the Java language would have to consider both the language itself and the libraries/frameworks built under it, the contributors to the language, both contributors to the language features and to its libraries/frameworks, etc… as well as the users of the language and its libraries, i.e. the developers that create Java applications.

The different types of contributors to the Java language. I would start by looking at the contributors and I would split them in 2 camps: corporate contributors and non-corporate contributors (*). Looking at the chief contributions these 2 types of contributors make to the Java platform I would say that the corporate contributors are mostly active contributing libraries and frameworks and that the non corporate contributors are contributing Java language variants such as Groovy and Scala (**). The split between contributors seems to mirror the split between the language itself and the libraries/frameworks being built under it, this is an important point.

It is important to look at the motivations of each contributor type for contributing to the Java platform: the corporate entities are mostly contributing libraries and frameworks that target specific problems with a wide audience while the non-corporate entities typically target problems with small audiences, such as languages. The size of the entities involved in a particular task is usually a good indicator of its audience and of the need for coordination (the bigger the number of entities, the bigger the need for coordination).
Corporate entities are more effective in projects where a certain amount of discipline and coordination is required (such as when defining the WS-* specs), while non-corporate entities are more effective in projects which do not require a large amount of coordination. The need for coordination between corporate entities is primarily driven by the number of these entities and the fact that these entities have different, sometimes competing, interests to which a common denominator has to be found.
Opposite to the need of coordination you would find most entities that created the languages currently running on the Java platform: These languages are initiated by individuals and are maintained by a single team that is usually pretty small.

One important thing about the contribution of corporate entities to the Java language in terms of libraries/frameworks is that these corporate entities need these huge investments in libraries/frameworks to be relevant in the future. Backward compatibility is very much needed for them in order to provide the stability required for continuing to make these contributions, this is very important to consider when thinking about truncating Java (see small section below) and when making language changes.

The mushrooming of JVM languages (***). You would expect that once the JRE got modified to allow easier access to different languages on the Java platform that the number of Java variants will grow very large, to the point of becoming unmanageable. This has not happened so far and if I were to identify the reasons for this I would first say that 1) there is an entry barrier for creating a language, you need to design it well and 2) the need to keep relevant investments made when committing to a particular language. The small number of languages used on the Java platform, in sharp contrast to the number of Lisp variants, is due to the fact that when a entity adopts a such a language it will make a commitment to use it in order to keep development costs down (you don’t want your web development shop to work 10 different languages, each needing a guru). The need to keep development costs down, which doesn’t exist in Lisp’s world because Lisp is used by academia rather than by everyday coders, put a ceiling over the number of languages running on the Java platform. I would expect this pressure on the number of languages running on the Java platform to exist in the future, new JVM languages will start being used only if they truly offer gains(****).

Open sourcing Java. Just like the mushrooming of Java languages the mushrooming of Java-like languages which branch out of the main Java language maintained by Sun didn’t happen (looking at the hundreds of Linux variants you would have thought that it should happen). I think that this is due to the exactly same thing that prevented the mushrooming of JVM languages - the commitment that using a language entails.

Truncating Java. The JRE got too big and even with today’s networks it is still a pain to download and install it. Split it up in OSGi bundles and spin them off the runtime (with Java AWT and Swing being the first victims). Keep in the runtime only what is absolutely necessary for the current enterprise libraries to work well (the collections library, threads, etc…).

Competition from Ruby. Ruby appears as a serious contender to Java, supposedly gaining market stare and mind-share. I don’t think that this will last long primarily because Ruby will not have resources for re-creating the current Java libraries. I also think that re-creating Java’s libraries in a different language is a waste of time. My opinion is that it is beneficial to learn Ruby in order to use JRuby and tap into the vast libraries currently available on the Java platform.

My opinion on Java’s present and future. I think that the current split in contribution to the Java platform (corporate contributors generating libraries, frameworks, products, etc… and the non-corporate contributors creating languages) is correct. Let each camp go forward in its own way, the corporate entities will continue to produce the specs, libraries and frameworks that we all use and that we all need to be relevant in the future while the non-corporate entities will work on creating new languages from which to call those libraries. There will not be a mushrooming of languages because the costs associated with using a language will keep the number of languages down. There will be some healthy competition from various languages that attain buzzword-status, but it will not last for long.
This would conclude my post. I admit that its subject (Java’s present and future) is very broad and covering it in one blog post is very hard but I have these opinions that I want to share with the world.

* By non-corporate entities I mean most small cohesive groups that grow around Java such as the groups maintaining Groovy, Scala, etc… By corporate entities I mean both large corporations involved in various specs (such as IBM, BEA, Oracle, etc…) and small corporations such as Spring Source or Red Hat. Also, the contribution of a corporate entity to the Java platform is not restricted to the contribution that entity makes in the open-source space, but it means all the code that entity has created in the Java platform, closed source as well. I know this definition borders somewhat on using the language rather than contributing to it, but I will keep it this way.

** I know that Java languages are also contributed by corporate entities with JRuby being contributed by Sun being one such example (Sun actually brought JRuby under its umbrella, JRuby being started by a non-corporate entity).

*** By JVM languages I mean languages different from Java that run on the JVM such as Grrovy, Scala, JRuby, etc…

**** Interestingly enough, Java platform languages would be a pretty good case of study for word-of-mouth advertising: some guy created Scala, another guy tried it and blogged about it, few more did the same, some other guy requested some features, etc…, before you know Scala is slowly being shoved into the spotlight.

Guys, I wouldn’t bank on people going on the web for cooking instructions, at least not now and not here in North America. I would definitely not bank on people searching your poorly-designed site in order to find out how to cook the food they bought from you while the water is boiling in the pan. I would bank on people reading the food package for this.

Cheers!

One interesting point is that legislation relating to price discrimination in one domain or another appears because it is demanded by the people rather than because it is demanded by some corporation or another.

I find that WL10’s console fails pretty badly the test of diversity: an application server would ideally need a console that can accommodate everybody from a developer testing its code to an operations guy managing it in production mode. The number of roles in IT is increasing and a product that is being used by various people in various roles needs to adapt to each of these roles. Sadly, WL10 doesn’t do it, it assumes that you need all the time all the bloat that it can provide.
You would expect WL10 to find a way to delegate handling the complexity of running servers to its users (such as the possibility of creating profiles that would allow someone to run common tasks more easily), but this doesn’t happen. I think this delegation is necessary in order to make WL10 more easy to work with for the growing number of IT roles that are using it.

The funniest thing that I see when I start up a server is the below screenshot detailing the settings for a particular WL10 server:

The red lines display the items I have never, or very rarely, interacted with. Why do I need to click on Federation Services for testing one of my EJBs? The number of unnecessary links that WL10’s console throws at you is probably somewhere around 80% of its “real-estate”. “Real-estate” being wasted away.
BEA should find a way to make its customers use its features (which I assume are quite useful) in a more efficient way. Throwing a ton of buttons, links, features at an unsuspecting user is not the way to go about it.

Well, the 2 camps are both right, PL-SQL is very good at performance, while Java is very good at modeling an application. We should use each platform’s strengths. So what would follow from this approach? Well, I think it is pretty obvious: create small PL-SQL procedures focused very well on their target and orchestrate them in Java. Avoid creating big PL-SQL procedures because one side effect of this approach would be creating high complexity in a language and a platform that doesn`t deal with complexity so well. Calling these small PL-SQL procedures amd assembling them into larger blocks from Java code is basically orchestrating them from a platform that handles complexity pretty well.

Do this and you will have a Happy New Year!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Disseminating knowledge is a process usually tied to the domain to which that knowledge is pertinent, to the organization dispersing the knowledge and the people that need to receive that knowledge. In this process you have to rely both on people as well as on systems in order to make it more efficient. Ideally, you would not swamp the persons with requests about knowledge, but rather put them in a position where they can guide people needing the knowledge to the appropriate knowledge resources.
Documentation is the primary vector of disseminating knowledge about a project*, it is the first point of contact between a person and that project and a point of contact to which someone will refer continously during the interaction with that project. It is important that we acknowledge this double role of documentation (initial point of contact and then continous point of contact) and that we structure it to accommodate both roles: a high-level view for initial contact and a lower-level view for displaying the internals of a project.

The knowledge about a domain “encapsulated” in one person depreciates slowly for a small period of time and then rapidly as more time passes with that person outside of the domain. For example, a person that is absent from a domain for a small period of time, during which the domain has changed, will be able to absorb the domain changes quickly, if it has been absent for a big period of time it may have problems getting back in shape. However, previous domain knowledge will help.

Storage and retrieval of knowledge is usually expensive. The more “fuzzy” the domain of that knowledge the more expensive it gets to retrieve good knowledge about it. In “fuzzy” domains you usually rely on people to get to good information. In “fuzzy” domains you will also need people in order to resolve conflicts in the interpretation of knowledge. It is important to identify these persons and set-up a communication channel with them.
The need for people in processing knowledge is probably one reason for the informal networks that get established in large organizations: sometimes it pays to know who knows something, sometimes it is the most efficient way to deal with knowledge. Getting to a person that knows something that you need to know is a process of discovery which will ideally be optimized.

Blogs can be used for delegating knowledge management to a person that is motivated to provide that knowledge continously. Bloggers are pretty attached to the subjects they blog about and are likely to follow these subjects and report on them efficiently (at least good bloggers should report efficiently). These reports constitute knowledge about the blogger’s subjects which are pretty precious. Another nice thing about blogs is that you can initiate conversations with their authors and get pointers from them, blogs act both as systems for disseminating knowledge and as people disseminating knowledge at the same time.
In an organization blogs could be used for finding persons that have knowledge about something and for contacting this person about that knowledge. We should be able to differentiate between blogs and to assign blogs to various knowledge resources.
I think there is a lot more to knowledge management in blogs than I can express right now, I will leave this for later.

The book Economics of Knowledge by Dominique Foray is a pretty good book on this subject.

* I will use the term project as the object to which knowledge applies.

As far as I know tech support is used for backing various open source projects such as a variety of Linux flavors, JBoss, Spring Framework, etc… Tech support relies heavily on the lack of knowledge of operating that open source package: you have some problems configuring JBoss’s transaction manager and the cheapest way to deal with it is to open a ticket with JBoss rather than try to jump in to the free code. Tech support is actually outsourcing the management of the knowledge required to operate a particular product and as such it feeds on the lack of knowledge about that product. If your product is big and hairy (think about a stack) but widely used you will make a lot of money from tech support. Tech support revenues grow as the complexity of operating that product grows because tech support is tied to the knowledge required to operate that product.

Tech support runs counter to light-weight frameworks, from their definitions they are light and easy to handle. Which makes you wonder about how would a framework championing light-weight development would be able to sell tech-support subscriptions. You will probably sell tech support to Fortune 500 companies which require support for most of the software they use (requirement which is probably a by-product of internal rules more than anything else) but anything else will probably be a tough sell. I understand a 20-person company buying tech support from JBoss for managing their application cluster better but I don’t understand why would the same company buy tech support for a lightweight container. By definition the usage of that container would be so straight-forward as not to make you need support, you should be able to solve a problem either looking it up on that framework’s forums or thru the documentation.

That’s why I am following Spring’s model with a lot of interest: the very goal they are trying to achieve (an easy to work with and non-intrusive framework) runs against the rationale of supporting that framework thru subscriptions to customer service. The steps that they are taking, such as releasing value added packages probably shows that supporting a light-weight framework is not very efficient and that you have to resort to a different model for funding it.
I would say that this new strategy is promising because these value-added packages will increase the knowledge required for efficient operation and implicitly the need for tech support.
Kudos to Spring Source for breaking away from an old model.

Later Edit: Tech support also relies on the complexity required for interacting with a particular product, be it an operating system, an application server, etc… The more complex the product, the bigger the need for tech support. Spring Source`s products so far were not complex to operate, on the contrary, their objective was to allow for transparent, non-intrusive development and ease of operation is part of this objective. With Spring Source move into value added packages they also move into more complex products (if you look at what Spring Integration is trying to achieve you will see a lot of very complex interactions between various products in the Spring Source stack). The more complex these products, the bigger the need for tech support. It will be interesting to watch how this will unwind.

What basically The nature of the Firm says is that if the costs of carrying out the activity on your own are less than the costs of delegating this to a dedicated party you should carry out this activity on your own rather than delegate it. You can see Coase’s theorem at work in the current outsourcing/offshoring trend: the transaction costs for outsourcing some operations are so low to the fact that a corporation can delegate efficiently various tasks to outside partners.

So how would this apply to software development? I would say that transaction costs demarcate pretty well what needs to be grouped together and carried out by one entity (which assumes a specialization of some sort) and what needs to be delegated to a different party. Well, this is what is needed for architecting a project: you need to determine how to group various classes, packages, systems together so that they minimize the interactions between them. The interactions between systems (or classes or packages) can be thought of as transactions: one class calls the other to do some work just like a home-builder calls the carpenter to install the hard-wood floor. By looking at the transaction costs between various components of an application you could determine what transactions (method calls) should be consolidated into component that handles them and what transactions should be allowed to exist on their own.

I would say that one big cost associated with a method call (or call to a package/outside system) is extensive orchestration between various classes (a method that calls 15 other methods from 7 different classes). If you have a pattern of method calls repeating over your code-base you should probably try to take a step back and consolidate these calls into a component that will serve the purpose of that particular orchestration. This way you will create a specialized component to which you delegate this orchestration, rather than replicating it multiple times. Efficient specialization of the components will incur lower transaction costs because it will push down on the number of interactions between components. For this you need domain knowledge. While designing a system you should look for specialization and for ways to achieve this specialization and once this specialization achieved you should look for ways to interact with these specialized components efficiently. If this is done properly you will end up with a system in which the cost of interaction between various components is low.

Another big cost for interacting with a different system is knowledge about the other system and class. The more knowledge you need to have about an external entity the more costly it is for you to interact with that system: you will need to be kept up-to-date with changes in this knowledge, there is a ramp-up cost, you can handle only this much knowledge, etc… It is a good thing to minimize the knowledge required for interacting with an external entity, this way a component can interact more efficiently with other components.
One corollary would be that you need to externalize as little as possible from any component and try to keep most of its sub-components private. One side effect of having too much of a component available is that you run the danger of a component coupling to it unnecessarily.

One cost for interacting with an external system is API coupling: the fact that you need classes exported by that external system**** in order to connect to it. If your system shares classes with an external system you will need a way to synchronize the shared classes with the external system. This is done usually either by having the external system having different versions for its points of interaction or by having a mechanism that publishes classes used for interaction. Either alternative is pretty costly, so you may try to keep away from sharing classes with an external system (though sometimes it may be impossible).
Transaction costs can be applied in project management as well: you should look out complex interactions between teams and try to assign them to a resource best dedicated to handle them (this resource could be a team-member, a well-defined process, etc…). However, in this case you should avoid assigning this interaction to a real person because you will run into scarcity constraints (i.e. you will run out of people) very fast if you keep delegating to real people. This is why these interactions are probably best addressed to a process.

I think that developing a software project while keeping an eye on the transaction costs is a good approach.

Later Edit. Transaction costs are very useful in determining the interaction between large components (systems, services, applications) and they are probably best used for managing this interaction. When you design the interaction of various systems the interaction costs (which are transaction costs) should be kept to a minimum for that interaction to occur without many problems.

* In this post transaction will be used for its economic meaning: i.e. it will stand for an exchange between 2 parties.

** If I am not mistaken Ronald Coase received the Nobel prize in economics for his work on transaction costs and this paper.

*** Please note that there are people that actually build their own houses on their own. The vast majority of them are in the construction business or have connections in the construction business so their transaction costs are a lot lower that for the ordinary person.
On a different note, the orchestration needed for building a home (calling the people that put up the jeep-rock after the 2/4`s have been erected, calling the painter for painting the walls after the walls have been put up, etc…) is a very simple business process without any intrisic value and prone to copying. No wonder that the builder itself makes a pretty small margin while the lion`s share of the profits go to the entity which owns the land…

**** You are generally sharing classes with an external system if you interact with it via RMI or EJB. Interacting with an external system via WS-* or REST is a lighter coupling.

I have become interested lately into knowledge management. It is a pretty interesting field that applies pretty well to software development.

Later Edit: In the ideal case a person could learn a bit about the domain by going over the APIs, in such an environment you could say that the domain knowledge is pretty well encapsulated in the code. Of course, this would not mean that the domain knowledge should only exist in the code, only that it is a desirable to have the code reflect the domain.

Very clever quote. The business domain keeps changing a lot more frequently these days, I think due to an increase of the number of stakeholders which increases the variety of requirements, so it is imperative to be able to make those domains changes easily. Being unable to make these domain changes because of technological reasons carries a pretty big cost.
This is why the technology should take a back seat to the domain. Ideally it would be abstracted so that it is decoupled from the domain.

Another quote that I liked was: you are given a domain when you start working on a project, you are not making this domain up and that your implementation should reflect very well that domain. The job of the domain modeler is to find out that domain and then represent it as faithfully as possible. The domain will change, but it not change in a radical fashion because this would imply that the underlying business changes radically and this doesn’t happen that often.
As I was saying before, the domain changes more frequently these days so it is better to have your application very well aligned with the domain so that changes in the domain can be translated into changes in the application very easily. If you cannot add a new feature because some framework doesn’t allow you to then you have a pretty big problem.

Another big theme in this panel was how to defer decisions till the last possible moment, which is a pretty neat things: it pays off to make decisions later when you probably have more information to base your decisions on.

All in all, a pretty interesting presentation.

It is interesting to see how will Facebook react to Open Social. Putting a widget both on an Open Social site and on Facebook is not extremely hard, you decouple the interaction between the widget and the host system from the services of the widget and you code 2 interactions: one for Open Social and Facebook (the fact that you have to code to only 2 different APIs which are quite simple helps a lot). Also, from what I understand you are basically coding to a container API, which probably doesn’t require a very complex interaction. Due to the low cost of creating cross-corporation widgets via Open Social Facebook doesn’t have a big room to maneuver, it cannot take the widgets developers hostage as Microsoft and various telcos have done it (the high cost of developing to a particular platform prevented developers from creating applications that would work both on Windows as well as on Linux). Sooner or later Facebook will have to face the facts and drop its platform altogether because it will become just a redundancy.
One smart thing that Open Social has done is to create an API which is very simple. When they created a API competing with Facebooks they have created opportunity costs for coding to Facebook. The fact that this competing API comes with a low cost of development means that coding to Facebook carries a low opportunity cost which means further that these APIs are not competing for developemnt resources (coders, managers, etc…), a very important thing if you want to gain mind and market share.

Later edit: I think I rushed ahead when saying that Facebook will have to drop its platform. Currently there is not much difference between the Facebook platform and Open Social, both basically let you deploy your application within a social network. Currently both of them are basically containers. But this could change in the future. One potential important difference is how these containers will give an application access to members using the social network on which it is deployed (users, advertisers, etc…). Access to members within a social network could be what would split the organizations backing Open Social (hi5, Orkut, Ning, etc…) because a social network is all about access and different social networks would need different access between members. Anyway, I will end this here.

Later edit 2: It looks like there is an explosion of social network platforms. It is interesting to see how this will unfold: as far as I know the costs of coding to a particular social network API are pretty low if the application is well designed (you are simply coding to a container API with whom your application has a pretty simple interaction) so it is conceivable that developers for social network applications will be able to code to multiple social networks at the same time. Or who knows, maybe developing widgets for social networks will go the way of developing mobile applications: a highly fragmented environment where the costs of coding to a mobile device/operator combinations are so high that the developer is taken hostage by that mobile device/operator. Open Social tried to provide an API to make developing to the whole social networking industry easy (an initiative which bears many similarities with Google’s Android BTW), but they may fail.

One recent development in the IT world is the emergence of the Software as a Service, or of services that are provided to someone running an IT operation. This is due primarily due to the drop in the cost of communication which allowed for closer partnerships.

The most important thing about consuming a service is that it allows you to decompose the interaction with the Service Provider. While previously you needed to purchase a big,fat shrink-wrapped product and then orient your application around it now you can pick from the offerings that a service has and choose the ones that bets suit you from a cost, QoS, etc… perspective. This, obviously, doesn’t prevent the sale of consolidated service stacks. (However, you should be careful when buying a service stack because you are also buying the relationships between these the services in this stack. These relationships will ideally be customizable easily).

The different economics (the fact that you are buying the service thru a subscription rather than with one payment) drives a different interaction: the Service Provider has a big incentive to take you “hostage”. On the other side the consumer of a service wants to minimize the costs of transition from one Service Provider to another in order not to be taken “hostage”.Taking a Service Consumer “hostage” is done thru features, but it can also be done by coupling your application to that service. You should avoid coupling in order to avoid being taken hostage. Features that are not implemented by competing Service Providers will probably be more expensive and will also take you hostage (funny how you actually pay more for being taken hostage).
One type of coupling between a Service Consumer and a Service Provider is thru APIs. The Service Consumer should abstract the interaction with the Service Provider in order to minimize this coupling so that they could switch to a different Service Provider easily if they need to.The service consumers have the upper hand in this game, if the interaction with a service is properly designed switching to a different provider should be easy.

Another type of coupling between your operation and the Service Provider is thru data: you need to keep control over your data so that you can move from one service provider more easily. When you are moving from one service provider to another you should make sure that you keep control over the data so that your data is not taken hostage by your service provider. Providing data to your service provider should be a process that should be very careful organized, you need to keep a tight control over it. You need to keep a very tight control as well over the data produced by your service provider: invoices of the things you have sold thru that Service Provider, the customer records that a CRM service provider has about your customers, etc… A service provider which cannot communicate to you the data that it has produced on your behalf should be considered risky, you may be taken “hostage”.
One particular type of data that you may be interested in is data concerning the service usage. Consider this example: one retailer outsourced its B2C site to a Service Provider. The site was recording a lot of traffic, but very few sales were taking place. Click-path-analysis (which is essentially scanning the data of how the B2C service was used) eventually revelead that the check out process was confusing. The service usage data is vey important, especially if the service is facing ordinary consumers.

The services that are outsourced to an Service Provider are not services which are strategic an operation but rather services that have been commoditized to the point where they don’t differentiate one operation from another. This implies that you should not outsource services that your company counts on for getting ahead of the competition. For example, if the customer experience is at the crux of your operation it doesn’t make much sense to outsource it to salesforce.com, because this would eliminate one important arrow from your quiver. However, if you are building an operation that revolves mainly importing various artifacts from China you may consider salesforce.com for managing your relations with your customers.

The service that you subscribe to should have some latency, the SLA’s regarding latency should be somewhat relaxed. Most of the services will be provided remotely from a different data-center, this should be considered. This would add more weight to the fact that you should not outsource services critical and strategic to your operation. You could mitigate this by delegating coarse-grained services to the Service Provider so that the interactions between your applications and the service you are using are not frequent. This would mean that it pays off to delegate more functionality to the Service Provider, as a result the service they provide should be easy to customize according to your needs.

When you are outsourcing a service to a Service Provider you are also outsourcing the maintenance, operation and the security of that service and its data. You should make sure that the Service Provider’s op team is up to the challenge (this point is so evident I didn’t think it should be stated).
Software as a Service will probably continue to drive the commoditization of various IT operations, SaaS is about outsourcing non-differentiating operations to an entity best equipped to handle them.

Another problem that I have is with the total lack of structure when it comes to project management. Rather than this thick Agile soup that tries to have an answer to everything we would be a lot better off if we had some IT-management patterns, similar to the GoF design patterns, I am trying to figure out what OOP would have looked like without those design patterns and it is not a nice picture. Who knows, maybe one day some guy will write a book where you could read something along these lines: “we had this team made up of 5 people, 3 junior, 2 senior, we had this project in this environment. We had some conflicts between the 2 seniors which were resolved in this way. We determined that the best way to tackle these problems was to split the tasks in this way, etc…”. This type of book, a collection of effective and widely applicable project management patterns, would serve IT a lot better than the latest iteration of Agile practices.

If you have a homogenous team then Agile process will probably work for you. The moment you run into a team where there are big differences between members and their skill sets you will see a different set of dynamics because you will want to make the most out of each member’s skill set. Unfortunately, Agile development doesn`t have an answer to handle this.

The focus on developers going as far as to declare that you only need developers in order to put thru a successful software project is pretty ludicrous, it’s pretty ludicrous and pompous for developers to say that you only need developers for a software project, it sounds like a second-hand shoe salesman.

Small iterations and continous release sometimes run you into troubles: consider that you are exposing your application to the outside world (in REST or WS or whatever). Let’s say that you make a small change in the way the app gets exposed, like you added a method or changed a signature. The only way you can release this to prod is by creating a new version of the same service so that you don’t break the dependencies on the current service. Well, versions cannot be created very often or you run the risk of developing Versionitis, AKA version management problems. This puts a lid on the number of times you can port to prod, you will be able to port to PROD only rarely in order to avoid going into a versioning nightmare.
Continous release probably applies to projects where you have no or minimal interaction with outside parties or where you have a fair degree of control over their environments.

Agile development is mostly a mirror of various development practices occurring at various development shops which happen to have their a very skilled workforce (think ThoughtWorks and the like). It would be a mistake to believe that all of these practices apply outside of these shops. My opinion is that you should watch Agile Development from a safe distance, pick what you think works for you, try to formalize it in order to get it closer to your needs and move on.

As always I don`t have the time to finish this post, it has been gestating for a while now, so I will release it as it is.

BTW, this post uses Bill Burke’s post only for displaying an example of annotation-creep.

Later edit: If you think about the way annotations are currently used (putting them all over your code base in order to insert a component into various frameworks) they can be thought of as some sort of cross-cutting concerns (they are cross-cutting to a certain extent). I wonder if using something like an aspect would not do a better job of expressing the various concerns that annotations are currently used for. Aspects are orthogonal to the code-base at the same time, a pretty neat thing.
Wouldn’t it make sense to WS-BA-enable your whole application by specifying a point-cut and then applying annotations against this point-cut? It would be a lot easier and it would de-couple your code-base from external concerns. I am not sure if it is doable (I could think of a solution, but it is not pretty).
I think that the way annotations are currently used is not 100% OK and that it poses some maintainability problems in the future. Putting information into annotations is partly a response to XML-creep (as Bill Burke pointed out in the comments) but it is a solution far from perfect. What we are dealing with here is the need to add some information to a component in order to plug it into a framework. Annotations, as well as XML, are not doing a good job because they address this problem (plugging the component into the framework) at the individual level, at the component level. What we really need is a mechanism that would let you plug a whole set of components into a framework easily (I suggested using point-cuts).

Later Edit 2.0: I think that adding querying capabilities at the code-base level (similar to AspectJ`s point-cut language) along with a mechanism to interpret these queries into Java would be the killer feature that Java lovers wait for. It would create a whole new way of developing: for example AOP would be implemented by joining point-cuts to an interception mechanism. A lot of the current mappings (in XML or in annotations) could be solved by defining a point-cut and then a mechanism to interpret this point cut. In the above case you would define a point-cut of your components that you want to WS-BA-enable and a mechanism for working with the components returned by this point-cut. Embedding a framework into an application would consist of creating a point-cut and then passing it to the framework in order to process it.
This would be application-level introspection and reflection and it would do away with both annotation-creep and XML-creep. I can`t dwell too much on it cause I gotta go to sleep.

Later Edit 3: In this post Bill Burke argues about using meta-annotation in order to de-couple your code-base from framework specific annotations. Adding a level of indirection between framework-specific annotations will free your application from framework-specific dependencies, this is for sure, but you will still tie your code to a representation (the representation specified by your annotations that serve as end-points for the original framework specific annotations). I am not sure that this is desired. I still think that your code-base should not be polluted by any external concerns.
It is interesting to see what will happen:
1) Will the world continue with annotation-creep and XML-creep for plugging their applications into various frameworks?
2) Will the world settle for a mid-way solution, like the meta-annotation solution suggested by Bill for the same problem?
3) Will the world try to solve this thru a different design?
4) Will the world decide that frame-work plugging is so costly that we might as well migrate to a different language/platform in order to reduce these costs?
It is all a question of supply and demand basic economics: plugging an application into a framework is currently a pretty costly operation. If the demand for this operation rises significantly (the number of times an application is plugged into a framework increases) it would make sense to make an investment (new methodology, new framework, etc…) in order to slash the costs. Either you make the investment or your language/platform becomes so costly so that it will prove cost-effective to move to a different language/platform.

When will we have track-back spam? And what shape will take the defense against it?

Later edit: Obviously track-back spam has a much higher cost than blog comments spam so this may make it un-profitable.
On the defense side: you are essentially using a third-party service (like Technorati search) over which you don’t have much control. You could set-up a filter on your site that would filter the results that this third-party gives you.
So, it looks like track-back spam is not the way to go. Mind you, if track-back spam would have been profitable we would have had it a long time ago.

Testing as documentation.
Testing is not documentation. Or rather put is poor documentation. Sure you encapsulate the desired behavior of your application in tests, but this doesn’t mean that you actually document it. One of the reasons that testing is not documentation because in order to get the desired behavior out of a test (especially a non-commented test) you need to reverse engineer the test. As you can imagine this is a very error-prone process. You could make your tests more readable by commenting them but those comments are the actual documentation and not the tests themselves. If tests were documentation then documenting the tests would be as documenting the documentation, a mind-numbing concept on its own.

Tests, and unit tests especially, are low-level entities, they are close to the code. Documentation requires views of the application at different levels, from the low-level (small pieces of code) to high-level (functionality).While you could put together a series of tests which would test the application at a higher-level (and you should probably do so in order to test your application at this level) I don’t think you can navigate the tests easily in order to determine which are high-level, which are low-level, etc…

Yet another reason for which testing is not documentation is the fact that documentation and testing have 2 different reasons to exist: documentation is about informing a developer about the application (and this includes a lot more than functionality) while testing is about making sure that certain desired functionality is being implemented correctly. Documenting an application requires that you express some concepts in the most efficient and less error-prone way so that a developer can pick up the documentation and understand it as easily as possible.
This process of encapsulating knowledge in order to make it available to someone is a very tricky process and the failure of various methodologies would be witness to this. My impression is that this is a process which requires a lot of research, particularly in knowledge management, and that the test as documentation solution advanced by the XP-guys is just another milestone on the road to greater confusion.
Some people may find test as documentation to work for them, but I think they are a very small minority. Documenting an application is a process that is particular to a development team and assuming that all development teams can be reduced to a common denominator (such as test-is-documentation) is a pretty big mistake.

Test-driven development.
Yet another interesting use of testing. First of all, do you really want to start new development with a compilation error (talk about starting on the right foot)? Probably not.
The biggest problem that I have with TDD is that it encourages you to write a test before you write a class this test is supposed to test. As you expect, you get a compilation error. However, one problem that arises is that you reference the target class in the test itself, and you do this by usually putting the target class in a dummy package. Once all is fine you re-factor your code base and the target class gets assigned to its proper package.
This is a classic example of designing after coding: moving the target class to its proper package is a design issue (assigning a package to a class is basically managing the relationships between your classes) which you get to do after the class is coded. I think that this is a pretty bad arrangement, ideally you design before you code.
I know that design is seriously looked down in the XP community and TDD is symptomatic of this attitude.

It also has a pretty big overhead and shows an impossibility in delegating a task to someone. The ideal software developer is working autonomously, has a fair understanding of the domain and a good grip on the technology involved in its day-to-day job. The ideal pair programmer is a developer which outside of its pair is like a fish out of water. You pick your ideal developer.

Some questions for pair-programming enthusiats: Pair programming shows a distrust in the persons paired up: when will you trust a person to work on its own? Does a software developer needs on-going tutoring in order to work? Isn`t it possible to teach that person how to write better code, to have some in-house workshops on coding in order to show some best practices? If you consider your workforce to be so deficient so that they need to be continously tutored don`t you think that you should probably do something about that? How do you prevent 2 developers working in pairs from re-enforcing the bad habits that they share?
The usual argument to defend pair programming is the following: programmer A can do task X in three minutes and task Y in twenty-five minutes, while the times will be reversed for programmer B. Put these two programmers together and they’ll be able to complete tasks X and tasks Y in three + three minutes. (This has been copied-and-pasted from Cedric Beust’s blog).
As you can imagine this is a pretty bad arrangement: at best it just displays that you don`t know your workforce well and you end up assigning tasks to the wrong persons, at worst you end up hiding deficiencies in your workforce: one person`s deficiencies could be hidden by its pair and you could end up assuming that each person of the pair reflects that pair`s capabilities. A pretty bad assumption.

All in all, pair programming is another software development practice that smacks of despair. I end this post here under the constraint of time. As you can imagine there is a lot more to be said about this practice, but I don`t have the time.

Iterations are mostly needed for dealing with uncertainties and change. They avoid costly and risky investments. Being able to deal with uncertainties opened the door to a greater interaction between the software developers and their customers, in some cases the software developers working with their customers at the company department level (a B2C software developer would work with the procurement department of a customer, a document management software developer would work with the HR department of the same company, etc…).
The main change in software development was the change in the way software is distributed, via the web with very low distribution costs. Once these distributions costs went down it made sense to create customized versions of your product and distribute it to a larger set of customers, rather than to the “target” customer. A shrink-wrapped software package could not be sold to 25 different types of customers each customer being able to demand its own needs.

After the distribution costs went down customization costs went down: you needed to customize an application easily and reliably. Abstraction played a big part in this.

Around the time the waterfall model started to crack at its seams and software development started to move towards iterations a transition from products (targeting only one type of customer) to services (targeting multiple and very different types of customers) appeared. Again, my opinion is that transition is due to the change in the way software is distributed and to the fact that you could customize it more easily than before.

My opinion is that there is not a fundamental contradiction between the waterfall model and the iterative models, they are 2 different ways of creating software which spun out of different environments.

Later edit: These practices are not actually mutually exclusive, they can be used by the same team on approximately the same project. A while ago when I was working for an ASP we had to re-write our B2B application, this decision was reached after a lot of study. We went over all our customers that were using our previous B2B app, we looked at all their requirements, all their needs, the way we implemented them and we created an abstraction of our application that was targetting 70-80% of the needs of our customers. This, along with the default implementation, took about a month and it falls pretty well within the waterfall model and the dreaded Big Design Up Front. Next, however, we changed the way we continued developing the application: we did small incremental changes as we interacted with our future customers and as we got more information about their needs. Creating the skeleton of the application could be done in the waterfall model: we had data to analyze (the data was our customer`s requirements), to model and to code. However, when dealing with our future customers we had no such data (we needed to extract it from the customers) so it made a lot of sense to switch to iterations.
As I said above, I believe that iterations work well with uncertainty and change.

THere are four essential freedoms that users should have: freedom zero is the freedom to run the program as you wish (there are programs that don’t even give you that much freedom). Freedom 1 is the freedom to study the source code of the program and then change it to make it do what you wish, instead of what the developer chose to impose on you. Freedom 2 is the freedom to help your neighbor. That’s the freedom to distribute exact copies to others when you wish, up to and including republication. And freedom three is the freedom to contribute to your community. That’s the freedom to distribute copies of your modified versions, when you wish. Up to and including publication. With these four freedoms, the users are in control, both individually and collectively.

What about freedom two, the freedom to copy the object and hand out copies. Well, there are no copiers for physical objects, so that’s totally meaningless. In science fiction in some cases, people have imaginged the idea of duplicators for physical objects. Like you could imaging a car copier. Imagine that I - imagine I had a car, and I parked it somewhere. I wouldn’t mind if somebody walked up carrying his car copier and plugged one end into my car and out the drew a duplicate car - with a different lock, I hope - and then he could pick up his copier, unlock his car, and drive away, and I wouldn’t even notice because by car would be unchanged. Why not? However, as a realistic question, it doesn’t exist. And likewise for freedom three, the freedom to copy your modified versions and hand out copies. Well, to some extent you can modify a car, and people do, but there are no copiers to copy you modified car with. That’s not the way cars are made, anyway. They’re not made by copiers, nobody has a copier. So this question “should hardware be free” is a - is a silly wiseguy question.

RMS should probably know that you can copy cars and that cars are copied ferociously (more often than not in China which among other things has very weak laws protecting IP) and that is a pretty big problem (the knock-off industry makes billions of dollars every year copying everything from cars to Gucci sun-glasses to medecine). You obviously will not copy the car thru a device, but rather thru a business process and what you copu is not the car, but its design (another type of intelectual property that RMS doesn`t seem to care about). RMS seems pretty much out of tune with the real world, he seems stuck in a world long gone. I do understand that issues that the current connectivity (bittorrent and the like) poses to artists, publishers and etc… and I understand that DRM is pretty much a bad technology trying to set-up defenses against file sharing, but I don`t understand why some thesis pioneered at MIT in the 60`s should have any effect right now. RMS has its GNU, he should be happy about it and let the world turn. It does so anyway.

Open-source is currently being used today mostly to create resource-pools dedicated to commodity software which happens to be mostly infrastructure (operating systems, databases, ESBs, etc…), this is pretty much what open source has morphed into. It`s nothing near the ideals pioneered by some people in the 60`s (including RMS), more often than not it is all about corporate partnerships. Open source has stopped being led by ideological reasons (the famous free as in freedom) in order to be led by economic reasons (free as in beer).

I`m off to the holidays!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

One other problem that the an UI management console puts forward is that it makes a developer (a person that usually is fluent in a few scripting languages) use a UI which is wasteful. It is wasteful because it wastes one of your most precious resources: the fact that your developer is a developer and that developers are familiar with scripts. UI solutions reduce your developer to the level of an ordinary user whose knowledge about an app server is reduced to the knowledge of various entities managed by your app server (servers, connection pools, datasources, applications, JMS queues, etc…) and the knowledge of clicking along. A developer would probably be more productive if it could take a look at its operational environment, determine various tasks which occur often and script up a solution to solving these tasks rather than clicking away. This is a lot better when you are perfoming composite tasks like adding a connection pool, exposing it thru a data source, adding this datasource to various servers and re-starting the servers to load the datasource. It makes a lot more sense to put up a script solution if you have to carry out these tasks across a cluster for the simple reason that you may forget to carry out a task for servers (or worse, for a series of servers).

Another problem with UI management consoles is that the error factor is higher: it is possible to click on the wrong button - `Do you really want me to shut down this server?` - and that it takes a long time to carry out a task. This it the problem with humans, they are slow and error-prone.

Now a bit about the languages used for creating management scripts. Given that most of the tasks that would be scripted are variations on the same theme (repetition occurs pretty often when you are managing servers) it would make sense to use a language in which you could define templates easily. An OO language (such as Ruby) would probably make a lot of sense. ant is good (and Weblogic lets you admin your app server thru ant tasks) but ant can only go so much.

One last advice that I would have would be to avoid duplication of scripts. I am saying this because Weblogic lets you create scripts for managing your server both in Python and in ant. You should avoid duplication and settle for one language.

Moving from an UI-based management console to managing your app servers thru scripts is a pretty dangerous task. If you are not careful and don`t plan this move well you may end up with a lot more problems. Think about managing your servers thru scripts as of programming your operational environment, the same problems that programming poses will arise in managing your operational environment. Bloat-ware, feature-creep. code duplication, etc… may rise to haunt the way you manage your app servers if you don’t write your scripts carefully. You would be well advised to try to do your best to avoid such problems, because they are very hard to solve once they appear.
What makes them `coding` your environment very hard to solve is that the stakes are a lot higher in this case than for your typical code blunders: your operational environment will depend on the way you `coded` it and the costs of refactoring these management scripts are huge: your entire cluster could go down if you are not too careful. A developer with a lot of experience would probably be needed to supervise this exercise.
P.S. It would probably not be a bad idea if in addition to providing a JMX interface to your application for managing your application, a JMX interface that can be displayed in an app server`s management console, you could also provide an ant task which would carry out the same operation on the application deployed on the app server remotely. I am not sure if this is actually possible (I think that one good solution would be the app server exposing the JMX interface thru an ant task) but it would be pretty nifty.

P.P.S I wrote this post because I had nothing better to do. Take it with a ton of salt.

One big problem that using a DL in a Java application is the exposure that the developers using it had to that DL. You will find that most developers will absorb a foreign language only so much, and will tend not to get too deep into it unless it applies heavily to their tasks. You will also find that getting to know well a DL requires quite a lot of effort, effort that most developers are not willing to put in. The correct use of a DL being related to the exposure to that DL it follows that you cannot really use a DL’s more sophisticated features and that you will also try to avoid developing complex constructs in a DL. The scripting example becomes very telling right now, because scripts are nimble little things which do a very small and simple task, I would say that this type of tasks can be coded by a regular Java developer in a language that it cannot master completely. I would even venture to say that because of high costs of adopting a DL DLs will never get beyond simple nimble things in the Java world. I am not so sure about coding the service layer in scripts though. Will the service layer dwindle in complexity to the point where you could script it up? I am not so sure, time will tell.

P.S. The costs for adopting a DL are the costs of adopting any language, in particular a domain-specific language (DSL). These costs are roll-back costs (you cannot roll-back a language out of an application so easily), training costs (the work-force needs to be trained in this language) and human resources costs (if you happen to choose a language which is not considered cool anymore you may end up having recruitment problems, who wants to work on something perceived as a dinosaur?)

This would explain why developers have the same reaction to DSLs as they have to DLs: a bit of fear laced with excitement.

P.S. 2 I wrote this post in a hurry, I think it shows thru.

Later Edit: It would be interesting to see that now with scripting languages moving into JEE applications whether JEE applications will be used by developers using scripting languages. If Rod was right in his assumption that the service layer could be script-ed up it is conceivable that a Java app could be picked by a Ruby shop that would modify its service layer as it fits its needs. I don’t see this happening because the Ruby people would have minimal control over the app in this case (in this scenario they would have control only over the service layer, even though in theory it is possible to plug Ruby business logic beans into the original Java app via Spring) and because it would require that the Ruby team has some exposure to a Java application deployment environment (Spring files, etc…). Again, it requires that a set of developers (Ruby developers in this case) has some exposure to the other language and environment (the Java application) with all the problems that this implies.
Later Edit 2: Today I read about an interesting case of scripting: JRuby GUI APIs. What I find interesting about coding GUI in Ruby is the fact that it lowers the barrier of entry to coding them to scripting, a coding style that is much closer to a graphic designer than your typical Swing event handlers. Who knows, maybe the future desktop designers will grok JRuby…

* AOP can be thought of as decoratoring unleashed, an advice applied to a class is a decorator.

Mapping a CRUD method to a business method is a pretty bad thing, if anything the service which gets exposed REST-style should have very clear URL-to-business method mappings. It is pretty bad when you advertise one thing and when you do another. Using CRUD methods masquerading as business methods is pushing round pegs into square holes, if anything it obfuscates the contract that the REST-talking application has with the outside world.
Another bad thing with using CRUD methods for implementing business logic operations is that it seems that you run out of options due to the poverty of the REST syntax coming from the limited number of HTTP operations that it maps to. From what I see you can only do 4 methods on an order GET,POST,PUT and DELETE. What should I do if I want to partially commit my order (send my order to a another trader that will fulfill it)? DELETE is taken, it cancels an order, what other method in the HTTP protocol should I use REST-style in order to signal the server that I need to partially commit an order?
REST is pretty limited as far as I see it and I see it fit only for CRUD methods. You can tout all over the place that it scales to Internet-size but fact is 4 CRUD operations do not scale to complex interactions. You can do pretty much everything using those 4 operations by twisting your domain to fit your impoverished environment, but you can do the same thing in COBOL, Fortran, etc…

Later edit: You could have probably implement the cancel order method REST-style by creating a new URL on which you would cancel an order thru a PUT operation like below:

But the problem still remains: you are trying to express a business contract using CRUD methods. Pushing round pegs into square holes…

Later edit:  When you remote an application the contract that this application exposes should be crystal clear. Having CRUD operations masquerade as business logic operations obfuscates this contract.

Anyway, my previous post was pretty theoretical, if anything, so I decided to provide an example of transparent development. My intention is to identify best practices that allow a developer to implement a new piece of business logic while making very small changes to the original code.
So this is the first post in this category (I hope more will follow). The example that I want to give is the following:
Suppose that you have a sequence of steps that is being widely used and that you need to modify this sequence only for a very small concern, how would you go about it. This is the example:
You have class AbstractTaxService that among other things implements the method prepareTradeForTax. This method is the sequence that I was talking about:
public class AbstractTaxService{

public method prepareTradeForTax(Trade trade){

initTaxService()
initTradeForTax()
enrichTrade(trade)
transformTradeToTaxFormat() // this will transform the trade in a format that the tax service understands

}

}
AbstractTaxService has 3 sub-classes that use the method prepareTradeForTax and which expand on it: CommodityTaxService which taxes commodity trades, EquityTaxService which taxes equity trades and FITaxService which taxes fixed income trades. Each one uses the prepareTradeForTax method as below:
public class CommodityTaxService{

public method prepareForTax(Trade trade){

super.prepareForTax(trade); // call the method in the super class
notifyCommodityTradeSource(trade); // will notify the source of the commodity trade that a trade is about to get taxed

}

}
public class EquityTaxService{

public method prepareForTax(Trade trade){

super.prepareForTax(trade); // call the method in the super class
notifyEquityTradeSource(trade); // will notify the source of the equity trade that a trade is about to get taxed

}

}
public class FITaxService{

// doesn`t override the method

}
All is well, the system has been running for a few months when this requirement comes in:
The FITaxService needs to call a  new method enrichFITrade before transformTradeToTaxFormat and after enrichTrade for a particular trade type. Oops. The sequence gets broken. The ugliest fix would be to copy and paste AbstractTaxService prepareForTax in FITaxService. Another fix would be to touch-up the original AbstractTaxService prepareForTax so that it calls only initTaxService and initTradeForTax and then change CommodityTaxService, EquityTaxService and FITaxService so that they will call the method transformTradeToTaxFormat in their versions of prepareForTax. But this is ugly, it requires massive changes and massive regression testing.

You would like to change nothing in the above configuration and yet somehow implement this new sequence in FITaxService. One transparent implementation would be the following:
Sub-class AbstractTaxService with FlexibleAbstractTaxService, a class that implements a flexible sequence of steps for preparing the tax trade. Have FITaxService extend FlexibleAbstractTaxService and use it in order to implement the new sequence of steps:
public class FlexibleAbstractTaxService extends AbstractTaxService{

protected String[] taxPrepareSequence = null;
public prepareForTax(Trade trade){

if(taxPrepareSequence == null)
super.prepareForTax(trade)
else
for(int i = 0; i < taxPrepareSequence.length; i++){

// go thru the sequence and execute methods using reflection

}

}

}
The new FITaxService is re-coded as follows:
public class FITaxService extends FlexibleAbstractTaxService {

public prepareForTax(Trade trade){
if(trade.isSpecial()){

// prepare the sequence for handling this special trade

taxPrepareSequence = new String[]{`initTaxService`, `initTradeForTax`, `enrichTrade`, `enrichFITrade`, `transformTradeToTaxFormat`};

 }

super.prepareForTax();

}

}

This way not much has been changed and the new requirement has been implemented transparently.

Some caveats about using this method: it is pretty expensive since it creates a new class, a pretty expensive operation. It would be a good thing if this new class would be flexible enough to cover future requirements, if anything it should be built with this in mind. Also, care should be taken, since this approach creates new classes it could be considered prone to class mushrooming.
The cost of this approach (creating a new class) should outweight the costs of other methods (building up a battery of regression tests in order to determine if the changes in multiple classes have not had a bad effect on their functionality) if this approach is to be taken.

P.S. Lately I have started to look at sub-classes as though they ar decorators which you can use in order to implement behavior while minimizing the changes done to a code-base. These decorators carry a cost (sub-classing which requires a new class) that should be minimized.

Remoting an application should be thought from now on as a transparent process which should be applied to an application transparently. This should have been done anyway, but the REST/WS split probably brings the point home a lot harder.

P.S. There is a big architectural difference between REST and WS, namely REST deals mostly with exposing state and its associated operations (CRUD) while WS deals mostly with exposing business logic. If I am correct REST can be thought of as a way to remote the data-layer, while WS as a way to remote the business logic. With REST the business logic layer is to developed on the client side, while WS gives it up-front. I think this is a pretty serious limitation of REST, the fact that you have to re-code the business logic in different environments is an effort which is not to be underestimated. But with REST you have control over your business logic and the possibilities for mashing up a different client are greatly enhanced.
WS can also remote the data layer, it would treat the data layer as another business concern that has to be exposed, so in this respect WS is more powerful than REST (it can implement REST-specific functionality if it wants to).
Again, the split between REST and WS is a split in development environments due to various market forces playing out in the IT landscape and not a debate about the efficiency of one style vs the other.

* I wonder if the famous get method of the REST architecture is symbolic of the relationships in these communities, it appears as though the Perl/PHP/Python communities can agree only on the lowest-common denominator. How more common and generic can you get than get?

So far I see events being used either as gateways into larger applications or the infrastructure for implementing simple, one-way, non-circular work-flows. In the infoq example the main hard problem resolved around circular flows: a parent sends a message to a child which sends it back to the parent which sends it again to the child. The solution advocated was pretty simplistic, if I remember the presenter suggested keeping track of the nodes a message has visited and do something in case it is visiting the same node twice, probably inserting the messages in an envelope which gets sent from node to node, hopefully in a transparent manner that would allow for different algorithm for detecting and resolving circular flows depending on the nature of the application using EDA.

Applications using event-driven-architectures are essentially message-passing applications and fall pretty easily within the distributed computing domain. Distributed computing problems are very hard problems and the interactions between distributed components are pretty hard to follow. I don’t think that we are currently in a position to use EDAs on a large scale because I don’t see the modeling tools and frameworks that could solve the hard problems invariably generated by a large number of nodes passing messages to one another (mind you, this need could be turned into a business case). However, most of the world would beg to differ: take a look at the example given here. It is pretty interesting to see a problem being solved by slapping the tag EDA on it. The article brings together a complex web of interactions, states EDA and assumes that this solves the problem.

Another problem that I see with EDA is the fact that you couple systems indirectly. With EDA you need the whole picture of the systems, you cannot really say that you will send a message to a system thru a web-service and you finished with it, it is possible that this system will get back to you in such a way as to create a circular flow. Given the fact that this flow spans different systems it could be very hard to implement algorithms that would detect the fact that a message has been sent multiple times because of a circular flow (such as in the infoq example. In the infoq example you could implement such a detection mechanism because you had control over the whole system). This scenario should be considered when implementing inter-system flows.

So I would conclude this post by saying that for the time being EDA is a technology which is at its beginnings and, as a result of it, which has a lot more fluff than stuff. It may evolve into something bigger in time, but it poses very hard problems once the number of nodes involved in a particular event chain increases. Also, as far as I am aware, there are not good modeling tools that can be used for applying EDA on a large scale.

Later Edit: From this article it sounds like TIBCO (a well-established distributed computing vendor) offers a solution called BusinessEvents for event processing. Judging from a podcast with some guy from TIBCO I would not be surprised if it turns out a pretty good product.

Later Edit: With Complex Event Processing and Event Stream Processing it looks like the industry is moving towards processing event streams at a single point in a network rather than trying to manage the events at the level of the whole network. Maybe this is what is needed by the market. It is also part of the push towards asynchronous processing that we are witnessing. Maybe I was too harsh on it…

2) Do this transparently by creating custom implementations for the interfaces which define your application’s business logic. You can use inheritance for accessing shared behavior between a new custom implementation and an already existing class and configuration for setting up the custom implementation easily. Again, configuration can be shared if necessary. This approach has the advantage that is a lot leaner than the previous one and is easier to read.
The above requirement would be implemented this way:
a) Change the pricing module to work off a pricing schema (this is similar to step d) above).
b) Create a mapping in Spring which maps regions to pricing schemas.
c) Create a mapping in Spring which thru scoping retrieves the user from the web app, retrieves its region (possibly thru geo-targetting) and populates the pricing module with the correct pricing schema.
As you can see the second approach is not intrusive at all, it doesn’t touch the code base at all. More importantly it doesn’t couple the pricing module to the user as it was the case with the first example.

One problem which appears with the 2 approaches is the one that arises when a support team (likely not close to the developers) is doing support for this application.
In the first case the support team has to know all the entities which are tied to a particular failure in the application which can be very hard to do. As the complexity of the system and the number of short-cuts increase the support team will have a pretty hard time to determine what went wrong where.

In the second case the support team can have a more subtle problem: since the customization is done transparently it doesn’t really know what could possibly go wrong in the customization (was a new region added in Spring.xml and mapped to the appropriate pricing schema, was there something else). In the first case it could have gone thru the maze of relationships which make up the customization, in the second it cannot do much if the error occurred in the customized part of the application, because that part is hidden away from it. The only thing it may have access to is the configuration of these customizations as well as to the functional spec of the customization.
It would not be a bad idea to think about what could go wrong with a custom implementation when you are creating it and create a trouble-shooter which could be available to the support team for identifying errors occurring within a customized implementation. This trouble-shooter would, ideally, be integrated seamlessly into the support process. In the above case the trouble shooter would have been exposing the mapping mechanism (which maps a user all the way to the pricing schema) to the support team, who could use it in order to determine if the mapping has problems (does a user from Alabama get assigned to Midwest pricing schema?) because what is hidden from it is this mapping. Once the support team determines that the mapping is done well it could look at possible problems with the corresponding pricing schema.

The idea here is that customization can be done explicitly (by using a complex system of relationships between various entities) or implicitly (transparently thru configuration). Either way, the support team should have access to it.

Later edit: It would be really neat if in the above example the user-to-pricing schema mappings would have been tested to the point where they would have been bullet-proof before putting in prod. Then the problems would have occurred in the various pricing schemas. A support person should have been enabled to determine quickly what pricing schema was used and resolve problems related to them.
Mappings management, or bindings management as I called them in a previous post, will gain importance as the relationships between various components will increase. These mappings will create a few problems on their own once they become too many…

I wonder if you could/should compose unit-tests (let’s say JUnit tests) similar to the way you compose applications. The new unit-test (the one that tests the application as it has been modified by the advice) could probably be derived from the 2 unit-tests, it would be interesting to see if you could compose the new unit-test with the same ease with which you compose the application. One problem with unit-testing an AOP-enhanced application that I see is the difference between the ease of use with which an advice can be applied to a code-base and the ease of use with which you can create a unit-test which can test the new application. Very likely the advice is applied in configuration while the AOP test is written in code, there is a pretty big difference in the time spent on creating the new application vs the time spent on creating the tests for the new application. I wonder if testing platforms should not be tweaked in order to allow for unit test re-composition.

It would be interesting to know why nobody is talking about this. Could it be because AOP is being used mainly for infrastructure issues which do no inject significant behavior into the application (I see unit-testing AOP-enhanced applications being very important for weaving business-logic advices into an application)? Could it be because of the difference in the costs of creating an AOP-enhanced application and testing it? Once again, time will tell…

P.S. This post is pretty much a collection of various thoughts that I have about AOP and unit testing. I realize that it is not very well structured, but I didn’t have the time to dwell too much on it.

I think that we should be able to diff and merge design documents such as UML diagrams. I would like to check out the Eclipse Modeling Framework and see how they save an ordinary UML diagram to disk (and I hope that they use an ASCII format and not a binary format). If an ASCII format is chosen for this it is possible to create a component that could diff and merge various UML blocks.
Now I have to find the time to inspect this. Not going to happen for a while.

Later Edit: I find it ludicrous that UML diagrams produced by one system (let`s say Microsoft Visio) cannot be consumed by another system (such as free-ware Poseidon). It`s like code written on JBuilder could not be opened on Eclipse. Pathetic. These walls should come one one way or another. The wide range of free UML editors are worth nothing if they cannot communicate with other UML editors.

I get the impression that client-server started to become main stream once the price for PCs started to drop and it became worth-while to start using these PCs rather than dumb-terminals. The alternative to rich-clients back then was ASCII-only dumb-terminals, it was obvious that the work-force would become a lot more productive if they switched to these rich-clients. I wonder why the rich-client was chosen rather than the thin client in the post-dumb-terminal IT environment. My impression is that this was done in order to minimize the traffic between the client and the server, at those speeds chatty applications were not possible.
Once the PCs prices started to drop the next barrier was the network. This took a while to go down, but once reasonable network speeds could be achieved this opened the door to a new concept: that of the application stored in a central place and of thin clients whose functionality it making requests to this application. One advantage to this approach is that by consolidating the application in a central place it obviates the versioning hell lived thru during the previous period. This was first massively used for pushing new applications (such as Amazon.com, eBay.com, yahoo.com, etc…) thru the web to consumers. Another advantage to the thin client was that it lowered the barrier for accessing an application by not requiring the customer to do anything (no download, no clumsy set-up process, no clumsy removal process, etc…). One potential problem with the rich client is the PC’s real-estate: you can fit and efficiently manage only that many applications on your PC, this puts a barrier of entry on the applications that can be run on a PC (how many would have installed an Amazon-type book browsing application on their PCs in the 90’s?). With the thin client this barrier goes away. I would argue that the drop in telecommunications costs drove the thin-client model and the economics behind it.
As telecommunications costs continued to drop the applications started becoming more and more chatty, leading to the point where the interactions behind them had to be managed efficiently. SOA comes to the rescue. End of story.

It appears that events in the CONSUMER’s market actually drove the ENTERPRISE IT environment. I guess the effects the consumer’s market has on the price of various computing devices (by effectively turning them into commodities) creates cost-savings which are recognized and used by enterprise IT. It would be interesting to muse over what the future factors of change for IT in an environment that has already seen massive price decreases. With rock bottom prices both in hardware and in telecommunications the next thing to decrease in price is probably software itself (and the open-source community is busy working on that).

There are a lot of complains that all these migrations (which are true paradigms shifts) are costing the companies lots of money. I agree with this, but you should consider that they also save a lot because otherwise nobody would do it. My .02$.

P.S. Another thing that I see happening right now is the re-emergence of the rich-client model. I see this as being correlated to the emergence of the mobile devices (another event in the CONSUMER electronics space) and their differences (reduced bandwidth and, more importantly, a very different interaction with the user) rather than recognizing that the rich-client model has any value.

P.P.S. I wrote this post in a hurry. It probably shows…

Later edit: If my assumption that hardware price changes due to demand in the consumer electronics space drove enterprise IT is correct it would be interesting to see what would be the next drivers. Changes in the price of software is a possible driver. It would be possible that changes in IT would be done on “merit”: i.e. a certain paradigm would be implemented because it is a more efficient way to write software rather than because it is related to a drop in the price of a certain hardware platform.

This post is password protected. To view it please enter your password below:

Password:

This post is password protected. To view it please enter your password below:

Password:

In Java it is a pretty limiting pattern: due to the single inheritance rule you cannot have the sub-class of a template (let’s call it “templated” class) extend multiple super-classes. Sometimes this is pretty limiting, I find it. One work-around to this problem is to change the template class so that it is not abstract anymore and declare all the abstract methods in an interface (the “life-cycle” interface) which the template class is using as a member variable. Then the interface would be implemented by a “templated” class which doesn’t need to sub-class the template class, which gives the “templated” class more freedom. The template class will be populated at run-time with the appropriate interface implementation thru an IoC mechanism. I find that the life-cycle defined in the template class can be extended as well by defining a sub-class of the “life-cycle” interface which could extend its contract. This way you can define more flexible life-cycles. (You can do the same thing with the traditional template design pattern as well but I find it to be harder).
I find that this arrangement pretty much does away with the template design pattern and makes your application more flexible by eliminating the need for sub-classing.

One interesting question would be why the above approach (the interface-flavored template) has not been used before and the template design patten has been used extensively? I have 2 answers to this:
1) The template design pattern pretty much casts in stone your life-cycle. Once defined the life-cycle can be changed only at great costs, and some times the stability given by this inflexibility is desired.
2) The drop in the cost of interface-to-implementation mapping with the introduction of IoC. Before IoC interface-to-implementation mappings were usually constructed with tons of factories (another rather inflexible design pattern) and with tons of overhead. With IoC the interface-to-implementation costs plummeted weakening the case for the template design pattern.

It would be interesting to see how the template design pattern will evolve. Ironically enough it is a design principle championed by the architects of the most popular IoC (Spring) so there is some backing for it. It gives some stability to the architecture by making its extension harder, this could be desired in some applications. The flexibility generated by the interface-flavored template comes with some management costs: you need to manage the relationships that will be spawned because of the use of this method.
Time will tell…

P.S.: One key difference between the “old-style” template and the interface-flavored template is the inheritance issue: in the old-style template you export the life-cycle thru inheriting from the template class (and you have only one chance at inheriting a class in Java) while in the interface-flavored one you export the life-cycle by defining it in an interface (which can be inherit from multiple sources). The “templated” class will not have a chance to inherit from another class in the old-style template, while it will be able to extend another one in the interface template. I think that because of this big difference the old-style template doesn’t scale well: its mechanism for exporting the life-cycle (sub-classing) cannot be re-used, I have not spent too much time thinking about it.
Another difference between the 2 approaches is that the ‘original’ template pattern puts together the lifecycle definition (the abstract methods) and the lifecycle execution (the code calling the abstract methods) in one place: the template class. The interface method splits this in 2 parts: the life-cycle definition is in the interface and the lifecycle execution is in a class using the lifecycle interface. Sometimes you may need the de-coupling given by the interface template pattern.

The main value of a black-box is that it has a contract that has been tested and accepted by the user.

It is nice to be able to provide alternate views of the same entity easily. This can be done thru alternative types. One nice things about providing alternate views of the same entity is that you can reconcile systems more easily.

Sometimes you can use sub-classing for dealing with tightly coupled systems. System A is using system B as a data repository and A is getting data from B using B’s DTOs. Unfortunately for A, A doesn’t transform these DTOs into its internal representation of the same data. B is about to make a massive refactoring of its code and this includes the DTOs and relationships between them. In order to have A use the same DTOs as before B decided that the new DTOs will sub-class the old ones, this way A will not have to change all the code that is using those DTOs (and which is massive). The new DTOs will sub-class the old ones, implement the old relationships and the new ones. If you think about it it is pretty bad: you cannot sub-class ad-nauseum and sometimes you may get a collision between the old relationships and the new ones. But it does its job.

I get the impression that MS will actually make a pretty important contribution to SOA, and that it will be in management where it will occur. SOA could be the means thru which MS will break into enterprise computing.

P.S. From what I read I see that annotations are widely used for creating pointcuts. This approach means that basically you are creating your pointcuts in Java code and not outside of it (in an AspectJ query or Spring config files) which is, to a certain degree, contrarian. I wonder if this is not related to the fact that currently you can manage your Java codebase a lot better than your spring.xml files or your AspectJ queries.

I concur with it, it is true: thru the use of RSS data is being exposed in a standard way and this standard is used for putting together different data sources. You could argue that RSS broke down the barriers where were separating content in different sites and that Pipes is the software which lets you combine these datasources in very imaginative ways. Pretty good. Except that I don’t think it will achieve the penetration touted by most people.
The biggest obstacle to making come true this vision of web sites as a series of datasources is the fact that Yahoo PIpes serves niche users and not breaking the barriers between various sites/datasources them. Mind you, breaking these barriers was not an easy task and keeping these barriers down relies heavily on unpaid abor (imagine what would happen to most of the Yahoo Pipes if people would stop pushing stories onto digg, book-marking items on del.icio.us, blogging about Lindsay Lohan, etc…). In a sense Yahoo Pipes lets you surf people’s past-times, but this is another story.

As I was saying, the problem is that there are way the needs to surf these datasources are very narrow. Taking a look at the numbers of times each pipe has been run would indicate that the pipes created so far are not that popular (low 5 digits are nothing today), which would indicate that they are not easily consumed. I assume that they are not easily consumed because these pipes are servicing very pointed needs, needs which are not shared by most people.
Looking at Yahoo Pipes as if it is a market you end up with the impression that it is an illiquid market: you cannot map pipe consumers to pipe producers easily. The main way of mapping pipe consumers to pipe producers is to make each pipe consumer create its own pipes by using an editor (which I must confess I have not used), the pipe consumer becomes a pipe producers and the market becomes liquid. Well, this is not happening because your average Joe will not be able to construct a meaningful Pipe, partly because it doesn’t really know where to look for content, how to filter that content efficiently and partly because it has no desire to learn how to chop-up XML (even if Yahoo claims to have made it braindead-easy).

Your average Joe needs an agent that would do that for him, that would bridge the gap between a pipe consumer and a pipe producer. Well, that agent would very likely not do it for free. So how would you put together average Joe and the agent that would service its very narrow needs? In an old post I was musing about how to service niche communities by creating mashups that would use other sites (Amazon, eBay, Google Maps and now Yahoo Pipes). An example of a Yahoo Pipe that could target a niche efficiently could be putting up a this Pipe that lets you find an apartment near parks, public libraries, produce markets, etc… You could take this pipe and put it up on a real-estate agency’s site and provide that real estate agency’s customer with something worth-while. The customer could do exactly the same thing on Yahoo Pipes, but it doesn’t know how to. Well, the real estate agency could provide this on its site and provide a better experience. The real estate agency coupled with the web-site developer that puts up pipe on the real estate agency’s site is the agent that bridges the gap between the pipe consumer and the pipe producers. I see this as a way to make the Yahoo Pipes market liquid.
Another way to make this market liquid would be create a market-place where pipe consumers would make requests for pipes along with prices and pipe producers would fulfill these requests and get paid, similar to Amazon’s MTurk. To be honest, I’m not sure if this is feasible, but it is worth noting.

Yahoo Pipes would be an interesting service. The web has been turned into a huge database where anyone with a browser can find what it needs. The revolution happened. Nobody cares.

Later Edit: In a certain way an Yahoo Pipe is a piece of code, a piece of intelectual property if you wish. Yahoo Pipes forces pipe producers to give their IP away for free (as far as I see). I wonder if this is not a barrier to adoption as well. If I make a good pipe I’d like to see some benefit coming out of it.
Another point concerning pipe creation is the large number of websites/datasources. Putting together an efficient pipe requires the pipe creator to know the data in these datasources pretty well, the barrier to pipe creation is not just chopping up data coming from different sources, but also picking up the appropriate sources for data. This is a far greater barrier to adoption as far as I see it because it shifts the effort of pipe creation from data transformation (trivial) to knowledge about data (not so trivial). My .02$

This is pretty interesting. I am not sure if you have heard the case for contract-first WS, one argument is that the web-service is essentially an interface (since it exposes a contract to the outside world) and generating it from an EJB or a POJO (as it was the case a couple of years ago) would basically imply that you are deriving an interface (the WS) from a concrete implementation (the EJB or the POJO), which, as you probably agree, is brain-dead. Later edit: This way of relating to remoting has some OOP ideological shades: you are basically forcing OOP concepts (deriving an implementation from an interface and not vice-versa) onto something that may not need to conform to these OOP concepts (inter-system communication in this case).
However, I am not buying this 100% anymore. The reason that I am not buying this 100% is that WS is basically a remoting mechanism among a few others (REST being its main competitor)and the the emergence of different application assembly platforms (frameworks, applications, suites that let you compose an application out of components that are exposed). One popular application-assembly platform is AJAX (which I happen to believe has been hyped out of proportions). AJAX favors small envelopes and REST is probably a very good fit for it, so chances are that you will need to expose your app REST-style. But, who knows, you may need to expose it WS-style as well because one system only talks WS and it needs to talk to you so you would need to remote your app in WS as well. I would say that the fact that your application needs to talk to different systems in different ways means that remoting an application is a functionality best done transparently (I am stating the obvious). It also means that you cannot fore-see 100% the way your app will interact with other apps and the contract-first WS are probably limited to scenarios when you have complete control over the interaction between that application and the other systems (which doesn’t happen very often).

On a side note, remoting an application is essentially an exercise in communication, and the way you do remote an application reflects the various stake-holders and their degree of influence quite a lot. The use of WS is pretty much a picture of the way your organization manages its communication channels: contract-first WS would probably fit a top-down approach in which the communication between various IT assets is spec-ed out before-hand, while WS used for remoting an existing application would imply an approach in which the communication is delegated to the team supporting that application and that is responsible for how it communicates with the outside world.
Not that the 2 are mutually exclusive: you can have a contract-first WS (designed top-down by an SOA architect) piggy-back on an currently running WS which has been remoting an application for ages. Another reason why these approaches are not mutually exclusive is that they have different scopes: top-down interaction is coarse-grained (you can control only high-level issues from the ivory tower) while delegating some degree of control to the IT team supporting the application is fine-grained (they can expose finer access to the application). Depending on what you need from that application you will interact with it differently and thru different channels.
SOA is a bit about control of the IT assets and it is partly a backlash against delegating control to the IT teams. A right balance should be found between what should be managed centrally and what should be delegated and to whom.

P.S. I didn’t sleep well last night and it probably shows. Please leave a message if something is not clear (as it is probably the case).

I find the Ubiquitous Language to be a very superficial way for acquiring and communicating domain knowledge. Getting trained in the domain itself by reading books is a far better proposition rather than jotting down words/expressions as you talk to a domain expert. Your conversations with domain experts will improve, this is for sure.

Later Edit: UB is not a way to acquire and communicate domain knowledge, but rather to define a domain. I find that UB, like other DDD artifacts such as bounded context, serves primarily to manage concepts and limit the costs of disseminating these concepts within a group.  These costs are confusion, mistaking one concept for another, falling behind in understanding the domain and the application, etc…
I was a bit harsh when I first wrote this post…

I never thought about this, but you can get locked into a particular piece of functionality. Functionality is usually tied to a particular business process, so you are basically locking yourself into a business process. Business process are transient and in today’s world in which change is the word of the day that particular business process will probably change in time as well. The coupling between your application and that particular business process will determine how easily it can be re-factored in order to align itself with the transformed (or new) business process. Process de-composition (emerging buzzword ;-), watch it carefully) will probably play a big role in taking care of all this. One idea is to break the process (or functionality) into blocks at various levels (from high-level to low-level) and then re-assemble these blocks so that they end up delivering the required functionality. In order to do this efficiently you need domain knowledge.

Tagging’s first problem is that it has a very low cost: pretty much everybody can assign a tag to every type of content, it will cost you just a few seconds. This low entry cost makes it prone to abuse and/or careless use (I think careless use is a lot more prone than abuse)*. Linking, on the other side, is a very expensive operation: among other things it carries a pretty big risk (the risk of driving people away from your site) and is limited by the real estate of your page (you can fit only so many links on your page, this generates competition between links). This high cost of linking results in relationships of higher quality (when you pay a higher price you usually check the goods a lot more).

Another problem with tagging is that it enables very simple relationships between pages. Basically, all the pages tagged “California” are created equal. You may refine the “California” set into sub-sets (let’s say “Los Angeles”, “San Francisco” which can be refined themselves) but you don’t get very far, more often than not you end up with a pretty short and thick tree (the length of the tree is the number of common attributes and its width is the number of web-pages sharing these attributes). Linking, on the other side, results in very complex graphs which are a gold-mine to who knows how to exploit them (just think of how much money has Google made out of PageRank). The relationships between pages is much more complex when you link them rather than when you tag them.
* I actually have a problem because of this low cost. I have been maintaining a collection of bookmarks on del.icio.us which is categorized by tagging for the last year or so. The low cost of tagging URLs has resulted in a lot of URLs being tagged incorrectly because I didn’t realize that over time these tags will add up pretty quickly. Right now I have quite a few tags that I’m trying to get rid of, but it’s not too easy. BTW, del.icio.us’s functionality is very limited.

And this concludes this post…

Java code:
public class PersonList{

protected List persons;
// some methods for manipulating persons (sounds pretty Orwellian, I know ;-))
}
public class MortgagePersonList extends PersonList{

// re-submits their mortgages application
public String resubmitMortages();

}

JSF code:

<!– set-up the PersonList managed bean –>

<managed-bean>

<managed-bean-name>personList</managed-bean-name>

<managed-bean-class>com.PersonList</managed-bean-class>

<managed-bean-scope>session</managed-bean-scope>

</managed-bean>

<!– set-up the MortgagePersonList managed bean –>

<managed-bean>

<managed-bean-name>mortgagePersonList</managed-bean-name>

<managed-bean-class>com.MortgagePersonList</managed-bean-class>

<managed-bean-scope>session</managed-bean-scope>

</managed-bean>
I was expecting that I could inherit one managed bean from another so that I could re-use the persons List transparently. No such luck. So I was left with saving the persons List in the Session in the PersonList bean and retrieving it from the Session in the MortgagePersonList object. I thought that I could use an environment-independent Session object, the only one that I could retrieve via FacesContext.getCurrentInstance().getExternalContext().getSession(false) was actually an Object which has to be cast down to HttpSession. It is pretty weird that the ExternalContext is modeled similarly to HttpServletRequest (just look at getRequestParameterMap, getRequestParameterNames, getRequestPathInfo, getRequestServletPath), it speaks a bit about JSF’s environment independence. It would be interesting to know the reasons behind not developing environment independent sessions (getSession should return an interface specifying the contract for a session and not an Object which can be cast down to a specific implementation like HttpSession), this could help when implementing dialogs.
One other thing that I hate about JSF is the amount of informal specifications, you know the fact that if your managed bean controller is returning a String than you should map that String to a navigation case and so on. It is pathetic, it would have been much better if this functionality would have been specified in a more rigorous way like defining it in an interface. But it looks like Sun didn’t feel quite up to the task of spec-ing out the user interaction (this is what JSP should be all about).
I really like the abstraction layer they created but it looks like it was fully finished. My 2 cents.

Wow. Does this mean that Spring’s transaction semantics are starting to compete with JEE’s transaction semantics for mind-share, and ultimately, market share? In what DSL (**) should you model your transactions: the old way by declaring them in ejb-jar.xml or Spring’s way by declaring them in spring-conf.xml? Is there a danger that one of these languages will become obsolete? Is there a danger that the current transaction semantics in Spring could be forked in order to better utilize a specific app-server’s capabilities? Is there a danger that these new transaction semantics be irreconcilable with the ones in the next Spring version?

In an earlier post I likened the transparent container’s capabilities to LISP’s, particularly when it comes to defining new languages (**). Well, the transparent container may inherit one bad thing from LISP: language mush-rooming. LISP diverged into so many dialects that it became eventually impossible to use as a common platform. More and more DSLs could be created for various use cases, which will range from the very specific to the very generic. I think this is probably where the effort will lie in the future, in managing all these DSLs, in separating specificity from generic behavior in an efficient manner. The future announces itself to be pretty messy…

* It is an interesting exercise to muse on Spring’s lead in TX APIs. If the features they advertise are needed - and I think they are needed and, even more, they are tested by the market - you would expect the JTA people to include them in their spec, after all the spec would be missing good functionality which should be included. If the JEE body will not include these features it will pretty much admit to being irrelevant in this particular area and would be handing the baton to the Spring people. Or maybe the JEE body doesn’t see these features as being generic enough or not having broad appeal. Who knows…

** Spring 2.0 was released with this nifty feature: XML namespaces. This opens the door to defining your own specialized languages (let’s call it a Domain Specific Language, DSL) tailored to your needs and plugging them into Spring’s runtime.

Anyway, what really interested me in this article was that this was a new format in which Microsoft was involved. These guys seem to be doing pretty much everything to break out of the Windows/Office cage on which their company was built. You see MS pretty much all over the map when it comes to standards, you start wondering what they are doing in all these bodies. I would assume it is part of their strategy of breaking into the corporate world by being involved in how corporations communicate one with another. You could argue that if you control the communication channels between various systems you could actually sell MS-enabled endpoints to service these communication channels in order to gain some foothold in the corporate IT environment, but this is just speculation.
Couple the involvement in all these bodies with the fact that they seem somewhat serious about opening their organization to the outside world (in a very controlled manner as always), you could easily see Microsoft morphing into a brand new company.
It would be interesting to watch it unfold.

* For however has any questions about the name, I picked it out of a hat. I need a name for it so I’ll call it this way. I could call it Spring, especially since Spring is the only container coming close to it, but I decided to go for an independent name.

* The tax service: When talking about an enterprise-wide service I usually give the example of a tax service that computes the taxes in a financial institution. Pretty much every transaction executed by all the systems of that financial institution in all the BUs needs to be taxed which raises the need for a taxation enterprise-wide service.

** A BU-specific service may evolve into an enterprise-wide service if the demand for its functionality make it necessary. One important difference between an enterprise-wide service and a BU-specific service is that the enterprise-wide service evolves pretty slowly, every change requiring agreement from a large number of actors.

Later Edit: Service management is essentially a study into the interactions between various various BUs in a large organization. These interactions will then be displayed in the interactions between the BUs’ IT environments.

Another heart has made the grade
Forget it, forget it, forget it
I don’t understand how the last card is played
But somehow the vital connection is made

I would give this example to display the benefits of code-based “configuration”: let’s say that you have a work-flow engine which models work-flows as a series of tasks, a work-flow engine that currently uses Spring for configuring the work-flows, for defining the flow sequences, etc… You become pretty successful and you sell your engine to a company that uses it for managing dozens of work-flows made up of dozens of tasks.
Now, your customer would like to have some business analysts create new work-flows on the fly (let’s say for simulating a business process), in other words they would need to create, test, discard, record dozens of work-flows looking for, let’s say, the business process with the lowest cost. It is obvious that the configuration file approach (declaring the relationships in spring.xml) is not going to do it because what is required over here is actually implementing a whole work-flow testing process. Now, your work-flow engine revolves around configuring work-flows, so one approach would be to develop or buy an app for testing workflows (any workflow, not necessarily from your application) that would be used by the business analysts looking for the lowest-cost flow and then plug this application into your work-flow configuration via Spring’s new code-based configuration.
The code-based configuration would actually act as a bridge between the work-flow testing system and the work-flow engine itself.
The whole theme of this example is that you are given the option to manage the configuration of your application how you want to (in the above example you wold manage the configuration of your work-flows by piggy-backing on a work-flow testing app), once this configuration cannot be met by the usual spring.xml file. This approach should be taken only when the complexity of the configuration raises to the point where it warrants this investment.

One last point about the transparent container called spring. It has been argued very well that Spring is non-intrusive in the application code. Well, it is intrusive, not in code, but in your application management process. If you are using spring for bindings management you are so dependent on it, you may not be able to continue functioning about it. But about this, maybe some other time.

The transparent container is a container that manages bindings between components and let’s you do it in the manner that you consider best suited to your needs: thru config files, code, etc… This manner is transparent to the component themselves.

Later Edit: I think that managing components relationships in Spring thru code is where the next cool things will occur in the Spring framework.

Later Edit 2: While I have not tried managing complex relationships in Spring thru code I think that this should be an easy process with a low entry-barrier. If this process requires a deep understanding of the Spring framework I’m afraid that it will never be used on the scale it could and it should.

Gotta go!!

Later Edit: While reading this post I started thinking about the duality of data vs functionality. Most developers spend most of their time dealing with functionality, I think they are missing out on a few interesting things. Another thought would be that if you have a problem regarding data the best person to talk about it would probably be a DBA, it probably has seen data more data than you can handle in a wide variety of formats. Just a thought.

// this is the method that publishes an item
public void publishItem(Item item) throws PublishingException;

}
public class DefaultPublishingBean implements PublishingBean{

public void publishItem(Item item) throws PublishingException{

// format the item
formatItem(item);// save it hard-disk
saveItem(item);

// syndicate it
populateRSSFeed(item);

}

// some formatting: adding menus, putting up banners, etc…
protected void formatItem(Item) throws PublishingException{
// format the item
}

}
DefaultPublishingBean is registered in Spring to implement the publishing bean:
<bean id=”publishingBean” class=”DefaultPublishingBean”/>

Now let’s say that you want to integrate the CMS with the forum. The integration will be very simple, at the end of each item you may or may not put a link which would create or update a thread on the forum related to that item. Part of this integration is a new publishing bean which contains logic interacting with the forum:
public class ForumPublishingBean implements PublishingBean extends DefaultPublishingBean{

// this method prepares the bean so that it doesn’t display the link to the forum
public void donotDisplayForumLink (boolean bool){
// set a flag
}// this method prepares the bean so that it links to the forum in ‘moderated’ mode
public void displayModeratedForumLink(boolean bool){
// set a flag
}

protected void formatItem(Item) throws PublishingException{
// if ‘do not display forum link’ mode is on then do not display the item

// if ‘display moderated forum link’ mode is on then put-up then display a moderated link

// else display the link at the bottom of the item.
}

}
ForumPublishingBean just needs to re-format the item, it delegates saving to disk, syndication, etc… to the super-class because this functionality didn’t change.
Right now the ForumPublishingBean replaces DefaultPublishingBean for publishing beans by the means of this bean definition:
<bean id=”publishingBean” class=”ForumPublishingBean”/>

OK, this was the set-up, now we are ready to start on the topic of this post. Everything is running OK, editors are happy,the community keeps talking on the forum when, all of a sudden, you hire Ann Coulter and Charles Kas an editor ;-), both in the ‘Politics’ section of your newspaper.
The result of hiring Ann is that a flame war starts raging on the forum any time Ann posts an article. The business starts getting fidgety and decide that Ann’s articles should not have the link towards the forum, this would limit the flames. This is business requirement #1: that Ann’s, and only Ann’s, articles should not provide a link towards the forum.
Next, Charles is not the flamethrower that Ann is, but he is not small potatoes either, as a result a lot of his items are getting flamed on the forum. The business decides that Charles’s articles should link to the forum in ‘moderated’ mode. This is business requirement #2.
Next, let’s leave politics aside for a while ;-), Vista is about to launch and the technology section of the paper is getting flamed on the forum by both Linux and MS die-hards. The business decides that for a while every article containing the words Vista should have moderated links to the forum and it doesn’t know when this rule will expire. This is business requirement #3.

So you have 3 requirements, quite different one from the other, very limited in scope and which should not pollute in any way either the CMS or the forum’s code base. Moreover, you are not sure in what direction these requirements will evolve, they are pretty specific. Do these requirements actually mean that you need to integrate the authors into the forum-CMS integration? Or do they mean that some sections, ‘Politics’ in this case, need moderation? It is pretty early to draw any assumptions from these business requirements because they are very poor.
One way to implement these business requirements without making any broader assumptions is to intercept the calls to the Publishing Bean and inject this very specific business logic. This could be done with 3 around advices as shown below:

import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
// this interceptor will check if the author of the Item that is getting published
// should have its forum link removed and remove its forum link
public class class BlockForumLinkByAuthorInterceptor implements MethodInterceptor{

private List blockedAuthors = new ArrayList();
public void setBlockedAuthors(List blockedAuthors){..}public Object invoke(MethodInvocation invocation){

// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
if(blockedAuthors.contains(cmsItem.getAuthor())){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// block this author’s forum link
publishBean.donotDisplayForumLink(true);// publish the item
publishBean.publish(cmsItem);

}else{

// go ahead with the original method
invocation.proceed();

}

}

}

// this interceptor will check if the author of the Item that is getting published
// should have its forum link moderated and moderate it
public class class ModerateForumLinkByAuthorInterceptor implements MethodInterceptor{

private List moderatedAuthors = new ArrayList();
public void setModeratedAuthors(List authors){..}public Object invoke(MethodInvocation invocation){

// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
if(moderatedAuthors.contains(cmsItem.getAuthor())){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// moderate this author’s forum link
publishBean.displayModeratedForumLink(true);// publish the item
publishBean.publish(cmsItem);

}else{

// go ahead with the original method
invocation.proceed();

}

}

}

// this interceptor will check if the section of the Item that is getting published
// should have its forum link moderated and moderate its forum link
public class class ModerateForumLinkByKeywordsInterceptor implements MethodInterceptor{

private List moderatedKeywords = new ArrayList();
public void setModeratedKeywords(List moderatedKeywords){..}public Object invoke(MethodInvocation invocation){

boolean moderateFlag = false;
// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
foreach(moderatedKeyword in moderatedKeywords){

if(cmsItem.getText().indexOf(moderatedKeyword) != -1){
moderateFlag = true;
}

}
if(moderateFlag){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// moderate this story’s forum link
publishBean.displayModeratedForumLink(true);

// publish the item
publishBean.publish(cmsItem);
}else{

// go ahead with the original method
invocation.proceed();

}

}

}
The above examples are woven into the application via this XML configuration:
   <!– The previous bean becomes the target of the advices above –>
   <bean id=”publishingBeanTarget” class=”ForumPublishingBean”/>
   
   <!– Register the 3 advices –>
   <bean id=”blockForumLinkByAuthorInterceptor” class=”BlockForumLinkByAuthorInterceptor”>
      <property name=”blockedAuthors” value=”Ann Coulter”/>
   </bean>
   <bean id=”moderateForumLinkByAuthorInterceptor” class=”ModerateForumLinkByAuthorInterceptor”>
      <property name=”moderatedAuthors” value=”Charles Krauthammer”/>
   </bean>
   <bean id=”moderateForumLinkByKeywordsInterceptor” class=”ModerateForumLinkByKeywordsInterceptor”>
      <property name=”moderatedKeywords” value=”Vista”/>
   </bean>
   
   <!– Set-up the proxy which replaces the original ForumPublishingBean –>
   <bean id=”publishingBean” class=”com,springframework.aop.framework.ProxyFactoryBean”>
      <property name=”proxyInterfaces” value=”ForumPublishingBean”/>
      <property name=”interceptorNames”>
         <list>
            <value>blockForumLinkByAuthorInterceptor</value>
            <value>moderateForumLinkByAuthorInterceptor</value>
            <value>moderateForumLinkByKeywordsInterceptor</value>
         </list>
      </property>
      <property name=”target”>
         <!– point this proxy to the original ForumPublishingBean –>
         <ref=”publishingBeanTarget”/>
      </property>
   </bean>

And this is pretty much it. The code above was using around advices, depending on the use-case you could have used other types of advices. Well, so we have implemented these very specific business rules transparently and without any intrusion into the application’s original code, which is not too bad. Now comes the hard part: when should you take this approach? (You know, the hard question is not how, but rather why ;-))

I think this approach should be taken when you are given some special business rules which you cannot generalize. In the above example it was pretty hard to determine what was driving the editing of the forum links: the author’s type, the words used in the CMS Item, etc… In a sense this approach buys you time during which you can watch the demand for a particular business requirement grow or stagnate and then decide whether to make an investement in handling it or not. In our example the interceptors that moderates forum links by author lets you see if this behavior is needed before modeling and implementing it: if you have 20 authors that need to be moderated then it probably pays off to model the moderation process, if you have only one it doesn’t make sense to invest time in it.

You could also take this approach when you don’t see the benefit of setting up a whole infrastructure in order to implement this business rules, when for example it doesn’t make sense to change the user management module in order to change the behavior of 2 or 3 users. In my experience I had a few cases where I needed to change a whole user module because one user was not supposed to click on a button, I wish I could have simply intercepted his calls and inject this very specific behavior.

You could also use this approach if you need to implement a simple business requirement that is set to expire soon. In our case we decided to moderate every CMS item that contains the word Vista in it. This business rule will probably be dropped in a few months once the excitement over Vista wears off. It doesn’t make sense to make a significant investment of code into something that will expire so soon, if you can get by with a small hack you may consider it. The beauty of this example is that you can shed this hack very easily when the need for it expires .

You should note that this example doesn’t scale very well and that it uses values hard-coded in configuration files. If this causes problems in your application, then probably you should not follow this example. Getting into scale problems is actually an indication of the fact that this interception framework is not servicing your needs anymore and that you should probably do something different about implementing this requirement.

One potential source of problems is the mushrooming of these business-related interceptors. Mushrooming code usually points to bad design, mushrooming interceptors points pretty much to the same thing. You should watch out if these interceptors are getting out of whack and once a certain threshold is met find a way to consolidate them, probably by making some generalizations about their roles.

One interesting thing about these interceptors is that they are not cross-cutting concerns, rather they are very specific about their target. I initially thought about using the title ‘Customizing applications transparently thru AOP’ but then I realized that I am not talking about aspects (which are cross-cutting concerns), but about very specific joinpoints.
Another interesting thing is that these are business-related interceptors and not infrastructure interceptors.

Well, this concludes this post. I wanted to have more time to talk about this, but I do not have the time. Feel free to drop some comments, I find this type of development pretty interesting.

P.S. I chose Ann Coulter and Charles Krauthammer because I needed an example of a business concern very limited in scope yet very important to the business. I didn’t have time to cook-up a better example so I just thought this one out. It is a bit off the wall, but again, I simply didn’t have the time. I chose some controversial authors at random in order to set-up a very specific business rule.

Later Edit As far as I am concerned I think the this approach is just another way of getting your development cycle closer to the business needs. This approach avoids building a whole infrastructure for very small business requirements and rather it lets you watch these business requirements grow before making an IT investment in them.

But this flexibility has an ugly reverse: the low signal-to-noise ratio which comes from many people using the same tags for tagging very different items. There are 40,000 words in the English language which you could use for tags and a probably a dozen million web-pages which are tagged in del.icio.us (del.icio.us hit 1 million users sometime ago, I assume each of them has at least 10 tagged items). Do you think that tagging can scale meaningfully at this level? You will try to apply various algorithms in order to differentiate the items being tagged (number of tags, frequency, time, etc…) but at the end of the day you will end up with very large datasets which need to be processed MANUALLY.

Which brings me to the next problem: the huge cost with false-positives: in order to register a false-positive (a page that has been tagged incorrectly and that doesn’t service your needs) you need to consume the information, to read it, and this takes usually a lot of time.
I gave up long time ago to retrieve good information by navigating my del.icio.us tags and trying to find common things with other people, the amount of information that I have to READ (which is a very expensive process) in order to find something worthwhile is ridiculous. I find that it is a lot more effective to keep reading from good sources of information in order to get an understanding of a new field rather than data-mine del.icio.us for my needs. It takes more time, but it is a lot less expensive.

One last problem with tagging is that it is a level playing field: a developer with 2 years of experience ranks on the same level as a someone who worked in, let’s say, transaction management for the last 15 years. If both of them tag different pages with the tag ‘transaction-management’ which one do you think is the page that better reflects the tag? Actually, let me re-phrase that: which page reflects the tag better for who? The pages tagged by the transaction-management specialist will probably be useful for someone working with high-level transactions and working on transaction propagation, while the pages tagged by the junior developer will be useful for another junior developer starting to learn about ACID. You could say that tagging works best in a homogenous environment where everybody expects pretty similar results from the same tag.

Tagging has been with us for a while, but so far it is used for pretty simple associations. I’m still waiting for the “wow” moment…

The lack of standardization in REST has some subtle side-effects, one of them being, of course, the fact that you have to work extra hours because in REST you have to code by hand the interaction with a different party. In WS you can look up the webservice, invoke a code generator and immediately generate the connection to that WS, in about 15 seconds you are ready to communicate with the other party. In REST you usually get a spec highliting the URLs which serve as entry points into the other system from which you derive the communication schema and you will go off to implement it. I am sure that quite a few REST developers out there have built their own generators for building the connection with a REST-enabled endpoint, but not everyone is proficient in such tasks. WS seriously beats REST when it comes to ease of use.

I like from REST the fact that the URLs can act as a level of indirection which in turn makes customization a lot easier. Small example: user A and user B use the same URLs from the same REST service, but user A has a platinum account while user B has the bronze account. Well, the application could serve more/better data to user A (more search results for example) thru the same URL by looking up the user’s account and making up all sort of decisions. In WS you cannot do this right now, you do not have the concept of a session yet (even if WS-Conversation is being spec-ed out) and the most implementations map a class or session bean to an end point and not the other way around (you need the endpoint mapped to a process easily and not vice-versa in order for that endpoint to become a level of indirection). The good thing about WS is that when this conversational feature will get implemented it will get implemented transparently.
One bad thing about WS is the fact that it seems to have set its mind on spec-ing out every conceivable form of communication. The spec is starting to mushroom and is being stretched in any conceivable direction in order to cover every God-forsaken issue. This is obviously a problem, but I think the market will solve it. It is a pretty simple case of supply and demand: if an issue has a big demand (in other words is considered important by the majority of OASIS members) OASIS (the supply side) will produce a quality spec covering it and the app server vendors will implement this spec. If the demand for such an issue is low, then it will not get as much attention from OASIS and it will either be put on the back-burner or a low-quality spec (reflecting the lack of resources dedicated to it and the poverty of use-cases due to little input) will be produced.
As far as the vendors are concerned they will concentrate on what is business-relevant and drop the rest of the protocols, simply because they will not have the resources to implement every spec that comes out of OASIS. It doesn’t make sense to implement things your customers don’t need only because they happen to be on a piece of paper that some committe wrote (it would make a lot more sense to have third-parties implement the most obscure parts of the WS spec and have the application server vendors plug these implementations into their app servers, but this is a side thought). I think that the vendors will focus on what is mostly needed (WS-Transaction being a good example) and then delegate the rest of the spec either to a third party, which could be open source projects under the OASIS umbrella, or the recycle bin ;-). I hope that the vendor community will take this market-based approach to the WS specs comig out of OASIS, it is a very flexible and efficient approach, and not get moored in a sea of meaningless and purposeless specs.

P.S. Well, you probably got used to this by now, but this post was also written in a hurry. I’m sorry, can’t do much about it.

Mismatches between demand and supply in a computing environment sometimes appear as the inability to scale. If in the interceptions example the inability to scale resulted in mushrooming of sub-classes which would basically re-implement the original methods with some interception logic around them, in the example of OOP done in C the inability to scale was the inability to have a large team of developers follow some coding rules that would have resulted an OOP-style coding. These 2 examples could not scale, or in other words as the demand (for interceptions) was increasing, the supply (the OOP environment in which these interceptions were coded) could not keep up. Inability to scale a behavior is basically a facet of a mis-match between demand and supply for a particular feature.
It would be interesting to see how you could address these mismatches. Looking at how the interceptions and C++ problems were addressed it would appear that it requires some large investment of some kind. This investment, which usually takes the form of a framework which addresses the problem, is strategic to a certain extent, since it will basically keep your costs down for a certain period of time.

P.S. I know that I am stretching some concepts over here, but quite frankly I don’t care. I’ll be continuing to explore this field as I find it pretty interesting.
P.P.S. I wrote this post also in a hurry, I may come back to it. I gotta go right now.

P.S. I wrote this post in a hurry, and unfortunately I could not expand on some aspects of this (the supply and demand of interceptions was something I wanted to explore more in detail). I would probably re-visit this post later.

P.S. I know I should have dropped this post in the ‘Stating the obvious’ category, but I don’t have such a category ;-).

Anyway, I am diverging. To get back to what I had in mind: one benefit of open APIs that stands out is reaching niche communities very effectively: Let’s take the group of “Chihuaua owners in the Hamptons” (wouldn’t you like to reach this group ;-)). Someone could pretty easily whip-up a site targetting them which would collect books from Amazon related to chihuauas, which could compare prices for the same book across Barnes&Nobles, Amazon, Chapters, use Google Maps for locating veterinary doctors, search ebay for latest chihuaua items, put together a discussion group on Google Groups, etc… All these APIs are pretty much open and free, what is needed is an entrepreneur that would realize that demand exist for putting all these services together (supposing again that this demand exist) and then sell very targeted solutions to these very small groups. Everybody would profit: the small entrepreneur has a business, ebay gets a cut on the items they sell, the ebay seller sells a product, Amazon sells another book they may not have sold, and Hamptons’s Chiuauas are spoiled rotten ;-). I think in the long run it is this very small entrepreneurs that would take the greatest advantage of these free APIs rather than the giants because they cannot create these applications which are very refined. To a certain extent you could say that the giants “do not scale in diversity” and that they “outsource” peddling their wares (books on Amazon, auctions on eBay, etc…) to niche groups or niche businesses to small-timers. In a very broad sense using small entrepreneurs which with your APIs target niche markets is similar to outsourcing your access to a very fragmented market.
In the example above I was using the community of head-hunters which could benefit from these mash-ups, there are dozens of other business lines that could benefit from these applications delivered with a surgical precision.

I think someone should write a ‘Software Development: stories from the trenches’ book, it is badly needed. I would buy one.

It looks like one major change that AJAX brought to development is concurrent access to state that was previously regarded in general as non-concurrent . It is interesting to see how the access to data in a concurrent fashion will be implemented. SEAM suggests that you should do this on the server, as according to this quote: “If you want to build an Ajaxified JSF application using ICEfaces or Ajax4JSF, you will run into all kinds of performance and concurrency problems if you don’t use Seam.”
Another way to implement concurrent access to shared state would be in the client: all the AJAX calls will be routed thru a manager that would determine (based on some configuration, etc…) which calls are to be serialized and which are to be executed immediately. BUT, AJAX is an asynchronous protocol, so you cannot serialize these calls at the client level because you do not know the order in which they will be executed on the server… *
So, it looks like we will have to deal with shared state more and more. The fact that this shared state is usually distributed and replicated (for handling fail-over, for example) makes it even more interesting. It would be interesting to see how this unfolds…
Later edit: I think you could serialize method calls that use AJAX calls in the browser by running them in a browser-based container that waits for a response from ther server to an AJAX call before making the next call. I don’t think I expressed it correctly though and I don’t have time to dwell on it…

There are a few ways to deal with this type of requests. The first thing to do is to refuse outright trying to deal with this legacy behavior, after all in 5, 10 years the new system will have plenty of legacy behavior on its own. Pushing forward legacy behavior is not a good strategy either, you are basically putting more work at each new wave of system renewal.

Sometimes this is not possible, for example, it is possible that no development is done on the old system within a reasonable time-frame.
One way to interact with this system is to isolate the legacy behavior that will get discarded away from the core of the applications and make it available to this system in some form or another. Abstraction would help very well in this regard, if the business process your application is implementing is well abstracted then you could create a customized implementation that contains the legacy behavior, implementation which could be discarded when the legacy system gets decommissioned for real. If you are working in a flow-based environment you could create a partition of your engine dedicated to legacy workflows and code your legacy behavior in a workflow deployed in this partition. Customized communication channels which talk to the legacy system pushing legacy behavior onto you should help as well for isolating behavior that is prone to become obsolete.

*On a side note: decommissioning a system should actually be the side effect of decommissioning a set of business rules (and not vice-versa).

And now the original post: 

The more I read about SOA the more I get the feeling that it grew out of N-tier architectures and that one its boosters was the so maligned EJBs.

When you are looking at an N-tier architecture you will probably notice the communication between various nodes providing various services to each other: this node gives you data on commodities, another one on users, another one on tax transactions, etc… The result of such an architecture is a proliferation of services on various nodes, services which have to be managed one way or another. In comes SOA to the rescue which manages the relationships between the services provided by various nodes thru governance (it is a lot more complex that this, but let’s keep things simple). It is the distributed nature of the N-tier architecture that created all these conversations between nodes, conversations which were then turned into services, services which started being managed with an SOA mind-set.
I think I am right when I say that SOA comes from distributed computing and that once the interaction between various computing nodes reaches a certain threshold it becomes mandatory to manage this interaction. If I was to look at the various distributed computing platforms I would pick EJBs as the ones which were dominant in the last 4-5 years, the ones with the greatest mind and market share. The built-in distributed nature of the EJBs has been frequently denounced as resource-hungry, backward, one-size-fits-all design. It would be ironic if this very nature played a part in the birth of the SOA paradigm, a paradigm that promises to start a whole new era in IT development.
What would be interesting to watch is if the various design patterns that the EJBs have spawned (business delegate, service locator, data transfer object, etc…) and that were sometimes denounced as band-aids to EJBs’ horrific problems will appear in one form or another in the IT environment which enables SOA. After all, the above designs patterns are mostly concerned with distributed computing.

Later edit: Communication between various systems existed long before EJBs, this is for sure. However, the EJBs increased the communication between various systems due to their distributed nature.

Another software design aspect which I tried to treat from an economist’s perspective is the managed environment (EJB containers, IoC containers, AOP-enabled environments, etc…). In the managed environment the developer is focusing exclusively on the business logic that has to be implemented while outside actors manage transparently issues such as concurrency, security, transactions, etc… Such a managed environment ressembles a vertically integrated organization which outsources its tasks (out-tasking is a pretty popular word these days, did you know that ;-)?) to a specialized work-force. Such an environment adapts itself to a changing environment more easily because it is the sum of some loosely coupled entities rather than a rigid entity whose elements are hard-coupled one to another. The modules that make up an application running in a managed environment would be very similar to the resources making up a multi-national: the modules/resources can be loaded/hired or let go according to the application/multi-national’s needs as these needs come or go. The result is a leaner application that responds to changes in the business environment more easily (again similar to a multi-national). The application would be a composition of various modules carried out by different actors, just like a multi-national is increasingly a composition of various tasks carried out by different actors. I’m not sure if I am seriously going off-road…

Anyway, I find mixing software design and economic theory fascinating and I will try to continue to treat various software design issues from an economic perspective. Comments welcome!!!

* In this posting I will use the term “hack” as with the meaning of gaining unauthorized access to various capabilities (not necessarily computer systems) due to the new complexity found in business processes. I am not including here the larger and more accepted term of tinkering with a “black-box” in order to gain understanding of its internal workings.

Later Edit: From this article (read to the end) it looks like process re-engineering is slowly emerging as a threat of disruption of normal business activity.

One the side issue of web development vs the rich client development:
I still hear developers scoffing at the HTML + Javascript combination while lauding Swing and its environment. Their primary irk at HTML is that it is pretty hard to code in JS all these dropdowns and these menus that interact one with another. They are missing 2 important points: One - there are quite a lot of JS libraries out there that do this for you (for free or for a small sum of money), and two - HTML freed the developer from working on the UI. I have not seen yet a Swing editor that lets a graphic designer work on a desktop app’s screen the way Dreamweaver lets it work on a page. HTML empowered the division of labor in a regular IT shop. If you think of it, a developer is not supposed to dabble in ergonomics, user interaction, usability and other concepts that pertain to UI development and that are second-nature to a graphic designer, it is the graphic designer that should be in control of them. What these developers actually needed but seemed unable to express was an event-driven development environment in which they could apply their expertise instead of struggling with clunky JS. Well, JSF just gave them that.
The benefits of HTML work along 2 axis: the division of labor and the usability of the tool used for manipulating the UI. This is one reason for which I think that rich clients are a bit over-rated, because most of the development environments for the new rich clients don’t come close to providing these benefits. AJAX (especially GWT) puts the developer back in UI development and provide it with a spartan interface. One tool standing out would be Flash. I’m not familiar with UI for mobile phones, I dabbled a bit in J2ME. I hope that these rich clients will not throw the developers back in UI development, but it sure looks like it. This is pathetic.
I didn’t sleep too well last night and it obviosly shows. Too bad…
Later edit: From this article it looks like the Eclipse RCP project aims to abstract the various rich clients platforms so that you could port the client easily across various platforms. If the Eclipse RCP manages to make the client portability easier it would obviate one of my points (about the need to move as much logic as possible to the server in order to accomodate a heterogenous client environment). I wish the Eclipse RCP a lot of success, especially in abstracting the mobile devices UI interfaces, those Midlets are begging to be let go.

So what would be side-effects of this architecture?
The first and most important side-effect is that the failure of a task doesn’t block the process, the rest of the items that have to be processed go on. Error reporting becomes more fine-grained, but also more difuse. You don’t have to scan a log that records every step of the process in order to get to the task’s stack trace because the node on which you have the log is dedicated to that particular task. One problem is that the error which is reported is particular to that task, you do not get the benefit of seeing the whole process dumped in a stack trace. In order to determine what went wrong you have to back-track from the current task to the previous task and in order to capture the whole information pertaining to that process instance. Trying to do this along a process spread across a cluster is not the easiest thing… In order to make this more easier you would have to propagate information considered helpful from task to task in order to report errors more meaningfully.
We would also have fine grained scalability. Once the process has been split into tasks and the communication between these tasks implemented you can scale each task independently of another. The result is that a computationally-intensive task cand be carried out over a larger part of the cluster than one that doesn’t require so many resources. The bad thing about fine grained scalability is that sometimes you need coarse-grained scalability. For example, let’s say that your organization plans to increase the usage of this process by a factor of 4, it is moving from processing 1000 items to 4000 items. You would basically have to scale the computational resources for every task 4 times and then test the new cluster. For a process with a large number of tasks it could be tedious.

Another bad side-effect is the difficulty with which you version such an architecture. If your process spreads across a whole cluster you may have to replicate the whole cluster and make sure that all the relationships between tasks are maintained in the new version. If you have a high tolerance to pain you may try to mix-and-match tasks from different versions: for example task 15 from version 4 would be related to task 16 from version 3 while waiting messaged from task 14 deployed in version 2, 4 and 5.

One good point about this architecture which is being brought up is the fact that you could reuse a task. Well, this doesn’t really apply because code-reuse is related to the specificity of a task: the more specific is a task, the less the potential for reuse.

To conclude this posting I would say that this architecture is a pretty good architecture when a process consists of a series of transformations, it decomposes the process meaningfully. Most of the side-effects are related to the length of the process, so it would help to keep the number of tasks under control. In order to better manage your process you may find that you may have to propagate additional information from task to task apart from the tasks require for carrying out their work.
If you have any other thoughts on this particular architecture or you have worked with such processes please drop a comment.

Obviously, this is a pretty bad error. The result was, among other things, a lot of copy and paste, a lot of redundant code and a lot of gotchas. This was pretty bad. I don’t know why, but the first thing that got into my mind was that this thing should be scrapped, which was obviously a very bad thing. Solely the fact that that code has gone from a half of dozen rounds of testing and has been deployed in production for a year made this code very valuable, because re-testing a new code-base implementing the same functionality would have been prohibitive. The correct way to look at it was not as at a pile of garbage, but rather as at some building blocks that could be used further. New contracts should have been developed for better alignment with the process which was getting implementing and the existing code blocks should have been adapted to these new contracts.

I was once listening to a JEE expert talking about a failed JEE project. He was talking with the lead architect and pointed out a flaw in their architecture. The architect decided to neglect the flaw and go ahead because time was running out. The end-result, according to the JEE expert, was that 12 months later the work of 40 developers was scrapped because of that flaw. I refuse to believe this. I cannot believe that the work done by those developers for one year could not be salvaged. I don’t know. I have seen projects going the way of the do-do bird but at the end various pieces could be salaged from the mess they were in. For example, a catalog module of an e-commerce application that never went live could be abstracted and plugged into another application (say, an application managing photo portofolios). Sometimes the application is way less than the sum of its parts…

Regarding the team in which you are working: odds are pretty good that you will not get a Rod Jonson to architect your application and that your team will be composed from pretty ordinary developers. Well, they are the people with whom you will develop your applications. Chances are they will make mistakes, chances are you will make mistakes, what is important is to deliver what you have to deliver. You have to find value in what has been developed and exploit that value as much as possible. (Looking at what has been developed from a functional perspective helps). Chances are whatever you are working on will change later and become obsolete. You have to deal with this change, and waking up one day to declare a whole code-base obsolete is not the right way to go about it.

Later edit: A REST application is probably best used for distributed computing where various nodes talk to one another. In this scenario the concept of a REST client and a REST server doesn’t exist, basically the REST client is emerged in the network the REST server(s) are running. You cannot have an external (at least I don’t think so) REST client, the REST client is similar to a spider that goes from node to node carrying out certain actions at each node and finishes by becoming part of the network. I think it is for this reason that trying to connect an external system to a REST-network fails, because the external system is to intimately tied to the REST network that it finishes by becoming part of it.
Comments welcome!!

Later edit: You can abstract the interaction with a REST-server and plug it in your application, except that it is awkward mainly because REST-ful behavior is very different from your standard interaction with a library that you know before-hand.
* I am not sure about this, please drop a comment if you want to talk about it.

Now, if we look at it from a “historical” point of view, the above doesn’t seem so bad for a first try. One cool thing about EJBs that resonates well with me is the fact that EJBs promoted the use of interfaces. One very good use of this would have been the abstraction of a business process behind interfaces, unfortunately this didn’t take off, most of the implementations were going the other way around, from concrete class to interface, XDoclet-style.
The “mistakes” that this operational environment made would be that it was advertised as “enterprise” and that it was too inclusive. Trying to address every issue important in enterprise computing on your first shot is prone to failure. The fact that this was done in a “democratic” fashion with each industry vendor pitching in his customers’ needs didn’t make defining the spec easier.

Some people look at EJBs as an abject failure. I look at it as the first step on the road to something new, a first step loaded with mistakes as all first steps are. Currently everything is running in a container, in a managed environment if you wish, which you get to define and tinker with if you wish. The lessons learned by working with EJBs provided valuable when the current successful containers were designed. It’s an evolution if anything else.

Let’s take an architect that spent the last 10 years architecting video games. It probably has its head filled with design patterns, UML diagrams, tricks, frameworks, etc… that enable it to do his job very well. Suppose that the same person is given the task of designing an application in the insurance industry. After collecting a mountain of use cases will it be able to see points of change in the application he’s building, will it be able to abstract the business process that it is implementing so that the application changes easily as the business process changes itself? I’d be surprised. That architect will go thru its collection of use cases and come up with a platform independent model and from it design the whole application. And he will probably fail to capture quite a few assumptions that the domain expert takes for granted and create a system that deviates from the original intent. It would have to adapt itself to the domain and then start to make some assumption about it in order to model it better.

UML can be used as a tool for communication between the business analysts and developers, but we should not expect too much from this tool. I think that the potential for errors and the noise is pretty big for this communication channel. Once things start to get complex these diagrams will not be able to translate efficiently the process described by the domain owners to the system architect. Instead of relying on various meetings with the business analysts, the architect should be able to understand the process, should be able to ask question meaningful to the business owners about the process that it is implementing and then it should start modeling it. It would help a lot if the business analysts would have some sort of a manual that distils the business and that the system architect can read before engaging with them.
I, for one, treat use cases and collaboration diagrams mostly as documentation needed for bringing a developer on board, rather than a way to pull information out of a domain.

As a final note I would say that the ability to adapt to a new environment is becoming a requirement for a developer. In order to be able for a developer to function efficiently across a wide array of industries that developer should learn how to adapt to a new business environment, and pretty easily. A developer should scratch beyond the surface of a industry in order to make meaningful decisions.

P.S. I’ll give an example from my experience.
I was designing an e-commerce application that had to support multiple customers and had to be industry-agnostic. What was the functionality that was most prone to change from one customer to another and which had to be abstracted first? Navigation? Layout? No. It was/is the prices. Every customer has its internal pricing structure (tiered, contract-based, rebates, etc…) that has to be reflected in the application. I learned it the hard-way, and it is a valuable lesson. When I designed a B2B application the first interface that I wrote was the interface for handling prices because this was one piece of functionality that I knew the customer cared about and that was prone to change. Do you think that this need ever transpired from the various interactions I had with the customers and various use cases I walked thru? No. The prices were the functionality that were the most prone to change and there was nothing out there to tell me that, most customers would assume that their price schema is what everyone is using. It was the interaction with quite a few customers that showed me that what is really prone to change is the pricing module.
P.P.S. In the above application the next thing that had to be abstracted was the interaction with their back-office systems, but it is a lot more obvious that it has to be abstracted.

Later Edit: Who would have thought this is possible? You know what I think? That the APIs are a commodity, what is not a commodity is the knowledge to use them, knowledge which is very important given the size and power of these APIs. It looks like RedHat is protecting that knowledge from becoming a commodity pretty well, but the manner in which they are doing it smacks of despair. You would have thought that the almighty “professional open source” model would not rely on hunting down individuals who piggyback on an OS product to make some money, but it looks like RedHat is looking at locking down that revenue stream.

Later Edit: I was reading this article on infoq.com about offline storage in AJAX applications. It looks like we have one more disconnected client type that will need the capabilities of SDO. Will anyone think about porting SDO to an AJAX environment? I hardly think so.

I am more interested in RSS’s behavior and the way you could apply to various problems.
One of the first uses would be to use RSS for broadcasting events in a group. Let’s say that you are a MySpace user and want to send a message to a group of people. MySpace could broadcast the message to your group by using RSS: each group member would have access to the group’s feed and any member could write to this feed. The result would be that each member could send a message which is guaranteed to be received by the rest of the group as long as the RSS server is up. Setting up this messaging infrastructure in a web application could be as easy as getting an RSS AJAX client and mapping it to the appropriate feed.
Let’s go to some exotic uses of RSS. The fact that RSS is a platform-agnostic broadcasting protocol means that it could be used for managing the interactions between various services in an SOA environment. To go back to the tax example: you have to apply a service (taxation) across a wide array of applications deployed on different platforms in a financial institution. You could have each of these application broadcast events (such as a stock have been traded, a commodity has been bought or sold, etc…) in an RSS feed and the tax program would listen to all these events and take appropriate action. The messaging infrastructure would be pretty simple to implement on both sides, I assume that there are quite a few clients written for various platforms.
You could use it for pushing deltas in a cluster if guaranteed delivery is not mandatory. A company could consolidate its messaging infrastructure using RSS: the email client would be replaced with an RSS client that is mapped to a specific feed, various groupware software could work off RSS feeds.

Various client-server interaction could be implemented easily with RSS: I wrote a desktop application that had to connect to a server and get the latest updates for catalogs, accounts, etc… This could have been done very easily using RSS. The changes in a catalog would have been published in an RSS feed and the client would have just had to connect to this feed and get its updates. Gone is the proprietary protocol which cost me a few nights, instead you get a tried, tested and true RSS client and plug it in your application.
An anti-virus or a patch manager could get a list of the latest virus definitions or patches and their location in an RSS feed and proceed with installation based on the information published in the feed.

I don’t want to be misunderstood. I am not suggesting to gut out MOM and replace it with RSS, I’m suggesting to consider RSS when you need an infrastructure for passing messages with a certain reliability rather than build it in-house. You could have various parts of your application talking to each other literally in seconds. I hold the conviction that code-writing is an error-proce process which has to be minimized as much as possible. In order to do this we have to start thinking in processes, we have to look at various technologies in an open way, realize their potential and use it.

Later edit: This site seems to be a good resource for RSS-related software.

P.S. Along with the Web 2.0 buzz some sort of a revival in UI is taking place. It looks like interacting with users is being thought about again. Martin Fowler’s GUI Architectures is worth checking out.
P.P.S. When writing this I kept thinking about the launch of Google Talk and the spartan user interface it had at the time. I was wondering how would an user of this product find its various services if the interface was kept simple.
P3.S. This article is concerned with UI management on desktop applications. I think these UI management requirements apply to web-based products but the implementation would be very different. A web application would be transformed into a portal that is easily to be managed by the user. For example, I would like to go to Yahoo’s page and have in the upper right corner some stock quotes, next to them press releases or news about those stocks and in the middle of the page general news. When I’m planning my vacation I would like the middle of the page to be a list of various trip packages rather that the news I usually have. I would like to put up these components and shut them down very easily.

* As far as I know Eclipse’s plug-and-play architecture is intrusive, you have to implement various interfaces, etc… Using Eclipse as a skeleton for building desktop apps would couple your application strongly to Eclipse unless you abstract the interaction between your UI components and create a bridge which will be used for having Eclipse boot-strap your application’s UI. This way you could use other plugable UI frameworks, such as Firefox, for assembling your application’s UI.

In the words of late Peter Drucker: People tend to concentrate on the word "Technology" when faced with the term "Information Technology". It is time to concentrate on "Information".
It is time to do something with this information. Handling it has become a piece of cake.

P.S. A lot of open source usually stands for developing infrastructure software; think about Sendmail, various web-servers, application servers, etc… So when you see that whole applications are getting open-sourced (think CMS, shopping carts, catalogs, CRM, etc…) it is a good bet that these applications are actually infrastructure, that they are the building blocks for something bigger and more valuable. It means that while previously you were supposed to work at a very low-level with database tables, triggers and Java code, from now on you may have to work with whole application stacks. Of course, not all development will be like this, someone will have to create these applications. But more and more of the work will consists of integrating these applications in order to create something of value.

To give an example for the above ideas: let’s say that you have a client that connects to a web-service. One way to implement this would be to use a code-generator (such as the one supplied with Apache Axis) and generate the stubs that connect to this web-service. Well, it is possible that the web-service would change in the future. In order to adapt to this change easily you could generate the web-service connection classes in a super-class (that would implement the infrastructure) and create a sub-class that would be called from your application (that would implement the business-logic). Changes in the web-service would result in changes in the super-class (which is not used by your application) and would not require big refactoring on your application.
Once again, I am not a big fan of code-generation. I was using the above as an example of the benefits of separating infrastructure from business-logic. You could use this separation for implementing generic behavior in a Template for example, but it is important to notice this separation and enforce it.

P.S. I wrote this post on the most horrible coffee I had in a long time. If it had an effect on the post itself I am sorry.
P.P.S. Separation of concerns resembles the division of labor as well.

P.S. The involvement of businesses in the creation and management of business-oriented JSRs benefits the IT corporations backing JEE a lot. The effort made by a business thru-out this process will translate in some form of a commitment to the JEE platform which would benefit the IT corporations involved and ultimately JEE. The JCP process could (and should) act as a communication channel between big-business and JEE.

CORBA is probably the first communication protocol that noticed the  variety of systems that were trying to communicate with each other and presented a solution to this problem. SOA is concerned at its foundation with this problem, but rather than solving it it assumes it is solved and moves on to determine what should happen once the systems do start communicating one with another.

Later edit: SAX is a parser really close to the metal begging to have some extension layered on top of it in order to make it more developer-friendly. The fact that no such extension exists or is not popular (I, for one, do not know of any) leads me to believe that SAX is not so widely used. Probably the reasons for which it was conceived (small memory footprint being the most important one) are not important anymore or the XML documents that are usually manipulated do not require a lot of resources.
Even later edit: It looks like the size of the XML documents and the number of formats grow pretty rapidly as SOA matures. A SAX-based OX mapper would not be a bad idea after all…

I have started my SCEA certification process and for the first exam I had to revisit the EJB architecture. I read EJB 2’s spec and at the end I was left in wonder. I could not believe that a corporation like SUN came out with such a spec.

The first (huge) surprise was the fact that there are interfaces that go against their contracts is a first (as far as I know) for SUN. For example, the MessageDrivenContext interface extends EJBContext and as a result it provides the MDB with access to its home. Except that an MDB has no access to a home since it is not referenced thru a home. Accessing the home thru a MessageDrivenContext would throw a system exception. I was in shock when I read this. You just have to wonder, what other methods are invalid in what interfaces? An API should provide a bullet-proof contract with no room for ‘buts’ or ‘maybes’. Well, this time SUN took a short-cut.

Another thing that surprised me was the breadth of the spec. The spec tries to address quite a few issues: transactions, security, concurrency, timer services, distributed computing, persistence and resource management. Addressing all these issues is obviously not trivial, and in this case the end-result in this case is a hodge-podge of an architecture. SUN made quite a few assumptions about the applications that will be built using EJBs and hard-wired these assumptions into the spec.
For example, concurrency is addressed in the simplest manner: all calls to a session bean are serialized. You cannot isolate calls based on state that is shared between calls as you would want in order to create a better-performing architecture.
Resource management assumes that 10 years after the first JRE garbage collection is still a big problem in enterprise computing.
In the persistence area: requiring that some tasks are carried out by pooled beans and others by pooled or active beans seems more like a hack than serious design, I don’t understand why persistence issues should be inter-twined with lifecycle management.

One thing that attracts developers to the so-called light frameworks is the fact that these frameworks do not make any assumptions about the applications getting built with them. Instead, they provide a way to assemble applications and bind behavior to various parts of an applications. They let you integrate outside components more easily into your application.

I think this is a more sensible approach to enterprise computing than trying to create this solution that tries to solve all the issues important to enterprise computing.

Finally, another failure of the EJB model is the collaboration between different technology providers. The EJB model was supposed to encourage application assembly: you would get some beans from a provider, put them together with some beans from another provider and create an application. I do not know of a successful application which got assembled this way.
However, the level of integration the lightweight frameworks have with outside technologies (such as the Acegi security framework) puts them in huge contrast with the EJB architecture. It looks like the IoC containers are much better applications assemblers than an EJB container.

The EJB spec leaves behind the image of an elephant mired in the moors of enterprise computing, trying to provide a solution for every conceivable problem, ending up with no clear answer to anything.
Looking at the picture that the EJB spec presents today it is not clear how another version – even the annotation-enhanced, transparent, lightweight version 3 – will fix the systemic problems that this bloated technology poses. I am skeptical that the on-going effort for refining EJBs will ever bear the fruit it was supposed to.

I noticed some tight-coupling for calls that returns HTML content which has Javascript due to not loading of JS libraries retrieved via AJAX. Some calls were getting some HTML content that required JS libraries for launching popups or DHTML manipulation. The HTML content came with the required JS libraries but the browser (IE 6) was not loading these libraries. As a result the calling page had to load the libraries that were used by the HTML content retrieved thru AJAX.
I didn’t like this because I wanted to be able to encapsulate my component in some form, shape or manner and then have it used without having to determine how are these components coupled and enforcing this coupling. It looks like at the current stage of development component encapsulation is not implemented in AJAX.
One problem this causes is that the component’s future development is pinned. You cannot add more functionality to it because the JS libraries which will implement this functionality will have to be imported in the pages using the AJAX component. You will have to document the dependencies between the AJAX component and the pages calling it, reload the libraries in the dependent pages and re-test them. Big effort…
Note: There is a work-around: getting the JS libraries and adding them to the document in a <script> tag. This would cause the browser to parse the JS required by the component. However this would require 2 calls: one to get the libraries, add them to the document in a <script> tag and another one to pull the component that uses these libs.
Also, it is possible that future browsers will determine if JS libraries are retrieved thru an AJAX call and parse them.

The concept of application on one page is a bit flawed. The problems posed by a browser’s refresh button will add to the problems of the back button and so on. The current AJAX client is missing some important functionality (it is not transactional for example) and probably due to the variety of JS implementations in browsers it will always miss this functionality. Drawing a boundary between what logic falls on the client and what falls on the server is an extra effort. App versioning and the browser’s JS cache pose some interesting problems too.

I wonder what will happen to an application written in AJAX and to its code base once W3C will put out HTML 4.01 (which among other things will separate the data from presentation and style)? Will these apps suddenly become obsolete? Probably yes.

The biggest drawback of AJAX is that AJAX is currently under-developed. Everything revolves around the return status of the http request. There are some libraries out there (such as DWR – Direct Web Remoting) and some frameworks are being developed but otherwise AJAX is under-developed. The plumbing is being developed by ear for now. It is possible that Microsoft will come out with their own plumbing for asynchronous client-server communication. It is likely the W3C will expand this technology as well. The danger that your current code base will grow obsolete is great. The danger that various independent AJAX implementations will emerge and require developers to code to 2 different implementations is also pretty big. Anyone remembers the Netscape-vs-IE wars?

The asynchronous nature of AJAX may pose some problems. Some ramp-up time is required for developing at ease in an asynchronous environment if you have not had to deal with asynchronous applications.

My opinion is that currently AJAX is pretty good for small tasks pertaining to presentation. Let’s say you have a result set presented in a list and you want to display detailed information about an object in that list without refreshing the page. You could do it in AJAX. You could share various presentation objects across a site with AJAX as well.
What you should probably not be doing right now is embark on large-scale projects revolving around this immature technology.

I think we could say that the language used in the XML descriptors of a package using declarative programming (such as an EJB XML descriptor, a struts-config.xml or bean-definitions.xml for Spring) fits the definition of a dynamic language.
You can change the behavior at runtime, you can create new classes and call them from different points in your program, you can even pass invalid objects to your program and find out that your program is not working because you didn’t XML-program (I have to admit ‘XML-program’ is an atrocious name) it correctly.

Declarative programming and the descriptors that enforce the contracts specified declaratively could be viewed as the place where a Java app would act in a dynamic way. The language of the application would be Java + various XML descriptors rather than only Java and this language could be viewed as dynamic.

So far declarative programming has been used mostly for infrastructure (EJB, Struts, etc…) but it would be interesting to see what results if it is being used for describing an application. I am thinking particularly about organizations that write programs that have to be easily customizable (such as a B2C shopping cart, a content-management system, etc…). It would be a good thing if I could describe this application in a declarative way so that I could describe and incorporate a customized business-process easily into the application. My app would be changed dynamically, customized behaviour would be created and passed to the appropriate components, etc… It would be written in Java and described and assembled in XML descriptors. It would be dynamic.

For example when the user presses the submit order button on an e-commerce site I should be able to specify in a declarative fashion that the DB will get updated, the credit card will be charged and the back-office system will know to send the product to the user and ask for replenishment. I want to add this and other functionality and turn it on/off at run-time.

To go back to the debate of Ruby vs Java, I think that Ruby will not get much penetration into enterprise development. The reason is that by now a huge investment has been made in JEE applications, an investment of the size of a few millions line of code that have been developed and, more importantly, tested and debugged. Assuming that you need to add some dynamic behavior to your product you could do so by applying declarative programming to your existing code base rather than re-writing the application once more in the language that happens to be popular. The debate of Ruby vs Java is a non-debate, it debates the current Ruby development environment and Java’s development environment as it was 4-5 years ago, i.e. Java running on its own. However, today most applications written in Java are usually run in some container that allows for loading behavior dynamically. Ruby vs Java + some frameworks would seem a more rational debate, however I have not seen anyone taking part in it.

1) Spring AOP lets you apply different behavior to the same class in different instances (actually you can do this in the same instance). It would be possible to download an open-source JEE application (such as a B2C shopping cart) and customize it without changing code resulting in different instances of the same app. For example, you would be able to declare that various steps in the checkout are transactional, apply different security requirements from instance to instance without changing the downloaded code.

2) The fact that annotation-based AOP binds behavior to code implies some development coupling: you have to develop or at least design the advice before you develop the class being advised. Ideally you would develop the advice independent of the code it gets applied to.

3) Annotation-based AOP creates horizontal relationships between classes that are very hard to change. All the classes sharing the same advice are tied together. Suppose that you have a Transaction annotation applied to you account manager, user signup manager and checkout manager. It would be very hard (and would require quite a lot of code-juggling) to change the app so that only your checkout process is transactional. These horizontal relationships (which as far as I know have not been integrated into modeling tools) should probably not exist at the code level, but at the application-deployment level. Cross-functional behavior should probably be applied at deployment time when the user’s requirements are ready to be implemented.

4) Annotations may degrade portability. One example would be the core EJB3 annotations for persistence. These annotations are platform independent since they are part of the EJB3 spec, however, these annotations are probably insufficient for real-life deployment because they do not allow for application tuning (such as data caching for example). In order to add caching to an EJB we would probably have to add server-specific annotations, and these server-specific annotations ruin the portability between application servers. With Spring AOP we could write an aspect that would piggy-back on the server-specific cache and advise the DAO with this aspect at deployment time. We could change the cache pretty easily by creating a new aspect for a different cache and advise the DAO with this aspect at deployment time.

5) Spring AOP creates a new role in the product’s life-cycle: the application assembler role. This role (probably best fulfilled by a business analyst which has a relation with the customer) would allow for making the application more flexible to customer’s requirements. A business analyst should be able to look at an application, determine what type of behavior should be attached to, or detached from its various parts and attach or detach that behavior. Supposing that this behavior doesn’t exist the business analyst should be able to have it developed in an advice that is independent of the overall application and advise various parts of the application with this advise.

I think that Spring’s proxy-based AOP is a better proposition that AOP thru annotations. Annotations promote relationships that are hard-coded and unless are specified by the domain problem they should be better implemented outside of code.