I went the other day to check ou The Onion, a satirical online newspaper that I haven’t read in a while. After leafing thru a few stories I was surprised to be prompted with thiw message:

You have arrived at your 30-day allowance of 5 free premium pages from America’s Finest News Source. If you enjoy our probing and analytical journalism and want full access, we ask that you support our hardworking reporters by purchasing a subscription for as low as $2.95/ month or $29.95/ year.

I was surprised that The Onion would ask you for paid membership after reaching a certain number of articles per month despite the fact that The Economist, New Times, Financial Times and other online newspapers have been using it for a while. A while ago, during the time that paid membership was written off to be replaced with advertising revenues, I alluded to the fact that the disappearance of prices was a temporary phenomenon which simply indicated an imbalance between the supply and demand of news.

The current price signals are indicating a rebalancing of supply and demand, a lot faster that I thought it is possible. The newspapers which managed to go thru this process will emerge with much greater strength. With the competition gone they will be able to operate at a scale which before was hard to imagine in order to become the first truly global newspapers.

P.S. The title is a tongue-in-cheek reference to Web 2.0 and whatever it was supposed to mean.

I got a feeling that out of the current companies which shape the way people use the internet (Apple, Google, Facebook and Amazon) Google and Amazon have the highest odds to continue do so in the future. Apple seems too concentrated on making devices and Facebook seems trapped into social media, for better or for worse. Google and Amazon lack a specific purpose, purpose which ultimately acts as a constraint, and as such have a much higher degree of freedom to adapt to the future.

This is not to say that Apple or Facebook are not profitable enterprises, or that their products are fads approaching their expiry date, only that they are bound by the very field they choose to dominate. Just a feeling…

I just read this article on highscalability.com and I have to say that I really like the idea of delegating the mining of your data to a another party specialized in handling huge data volumes which can in turn make it available to smaller players.

All the best to the DataSift team, they are building a terrific product!!

Une journée triste…

I read today a very interesting post on Dan Prichett’s blog on the way the IT architecture of a large organization is affected by the structure of the organization. For a while I have grown convinced that most of the decisions regarding IT are determined by the underlying organization, but I have not seen it stated so clearly. It is a pretty important statement because it identifies clearly what influences IT architecture the most.

However, I don’ think that the solution proposed by Dan Prichett to correcting the IT inefficiencies can be applied easily to any organization. I see the IT architecture driving the structure of the organization, rather than the reverse, happening mostly in IT-centric companies (*). I don’ see an insurance company or a distribution chain changing the way they do business solely for the purpose of enhancing the efficiency of IT. For a typical company IT is a sunk-cost rather than a strategic asset (**), restructuring a company in order to minimize this particular set of costs, possibly at the expense of other concerns, does not make much sense.
Adapting the IT infrastructure to the organization, and not the reverse, will be the norm for the foreseeable future. In most cases the best that IT can do is to try to reduce costs by automation of various tasks across the organization rather than try to define the organization.

Later Edit: Another reason why aligning the organization with the architecture (rather than the opposite) does not apply all the time is that refactoring the architecture will imply re-structuring the organization in order to align it with the new architecture. The already high costs or re-architecting the IT infrastructure may become prohibitive.

* I would call an IT-centric company a company that proposes services built around IT or that derives a  competitive advantage from its IT infrastructure. Examples of such companies are eBay, Amazon, Google, online shops, online media companies, etc… In these type of companies we may derive the organization from the architecture of IT systems because in doing so we enhance the competitive advantage of that company.

** For IT-centric companies this does not apply, in their case IT is a strategic asset.

I just finished reading this article by Jean-Jacques Dubray and I am left with the impression that the author blows out of proportions some IT developments.

The article showcases how the Massachussets Department of Transportation decided to open up a bus schedule data feed and found out that within a short period of time a number of client applications started using it. What is really interesting is that these applications provided basically the same functionality across a large number of platforms (iPhone apps, web-site, SMS and IVR systems). Basically MBTA delegated the development of client applications across multiple platforms to parties best suited for doing so.
A pretty neat division of labor emerged: the management of the data feed is carried out by MBTA while the client apps are developed by people at large.

Interesting as it may seem this development is neither new (mash-up have existed for quite a while) nor the beginning of a revolution in IT. This development is just the creation of an eco-system of applications built around MBTA’s data feed. I don’t share MBTA’s enthusiasm over it because delegating the development of client applications to benevolent third parties undermines the reliability of the bus schedules delivery. While this approach did allow them to cover a large number of client platforms without a significant investment it has also created a number of applications which seem to be nothing more that a hobby and which may be taken down without any notice. The eco-system of client apps will evolve over a certain period of time before settling down, during this period the delivery of bus schedules will be as fickle as the applications in charge of it. I am not sure that this is what MBTA had in mind when it opened its data feed to outside parties…

All in all, an interesting experiment in the relationships between various applications, but not a tipping point for the IT industry.

This presentation on domain event driven architecture by Stefan Norberg is a must-see presentation on loosely-coupled distributed systems. The solution of implementing cross-cutting concerns over a large number of heteregenous systems by having these systems issue domain events is one of the most elegant I have seen.

A point that I found very interesting is that systems must communicate via events and not via commands. For a system (master system A) to issue a command on another system (executing system B) it needs to know the context in which the command should be issued on system B. Requiring system A to have knowledge about the inner workings of a remote system in order to be able to carry out its tasks is typically how hard-coupling between systems starts.
The solution outlined by Stefan Norberg is to split the business logic for a specific domain into gathering data required for executing a certain task, transforming it into events and issuing events (perfomed by multiple systems), channeling the events to the system implementing the domain (typically performed by an ESB) and the implementation of the domain (performed by the system which receives the events from the ESB and uses them either for creating domain-specific commands or for creating the context in which domain-specific commands are executed). This partition of tasks scales very nicely as the number of systems and domains grow.

Another point that I found interesting was that this architecture requires the delivery of events in the order they were created. Some events are used for creating the context in which the events which are transformed into commands are running and therefore the events that create the context need to be processed before the events which are transformed into commands.
Let’s give an example. Suppose that the systems involved are a user management system, a bet processing system and a system managing user loyalty. Let’s say that we have this use case: user creates an account and then bets 5000$ on a horse race. For the sake of argument let’s assume that this use case involves the generation and dissemination of 2 events: UserCreatedEvent and UserPlacedBet event. Now let’s suppose that the system managing user loyalty receives the 2 events and is ready to process the UserPlacedBet event by checking the size of the bet and carrying out the appropriate command. Let’s say that for this command to the executed the user must exist in this system’s database. It follows that the UserCreatedEvent should have been processed before the UserPlacedBet event in order to create the context (in this case the context consists of relevant user data) in which the UserPlacedEvent could be turned into a command and executed.

A presentation which covers the use of domain events from a large number of angles (enterprise architecture, management, physical implementation) and which should be watched by anyone interested in large scale distributed systems.

“Growth now depended more on the operation of markets, something that sits uneasily with the institutional inheritance. Innovation-based growth  is risky, uncomfortably so for security-oriented European societies. It responds to financial incentives, which is difficult to reconcile with the value Europeans assign to earnings equality. It requires continous reallocation of labor resources, which is at odds with the importance they attach to job security”.
Barry Eichengreen - The European Economy since 1945: Coordinated capitalism and beyond.

A pretty good picture of the difficult terrain that Europe has to navigate.

Watching Eric Evans explaining how to incorporate agile methods in domain driven design gives you an understanding of some of the differences and commonalities between domain driven development and agile methods.

Agile methods grew out of the fact that up-front design which involved months of analysis was getting out of synch with the world which required faster development cycles and the need to adapt to a world changing at high speed (*). However, the responsiveness of agile methods and their focus on the next iteration limited development the scope at which development is carried out and eliminated the design phase out of the development process. The result were applications for which development slowed, if not even stopped, once the application reached a certain complexity.

Eric argues that in order to fix this problem we need to bring domain driven design back into the picture, but different from the top-down, process-heavy manner in which it was conducted before. The main reason modelling an application iteratively is that the development of an application is a learning process, a process of discovering and annotating a particular domain.

Some agile techniques such as the ability of reacting swiftly to changes and to be able to perform significant changes late in the game are presented, but they seem to me to be liabilities rather than assets. Performing a significant change late in the game is a very expensive operation since it is essentially a re-organization of the concepts encapsulated in the model followed by the confusion generated by the dissemination of the new model; such things should be avoided at all costs. Rapid changes in the model are also bad since they can create confusion.

One agile practice which I think benefits domain driven design is the break-down of the design process in iterations during which new domain concepts are inserted into the model. It is very important that these iterations also make sure that the model stays in synch with the domain in order to avoid late, massive and expensive domain re-factorings. In order to do this Eric outlines a series of diagnosis measures used for detecting when the model is straying away from the model.
While these measures are necessary what is missing in this process is a way for the domain expert to validate the model. The knowledge transfers have no feed-back loop from the domain experts. The developer seems the sole owner of the model, ideally the domain expert could become more engaged in the definition of the model rather than a passive disseminator of knowledge.

Modelling brand new domains or domains in a state of change will probably always be error-prone because the development of an application in a such a domain is a discovery process for all the parties involved (from users to business analysts to developers) whose end-result is defining the domain. In such an environment probably the only way to define and disseminate domain knowledge is to set-up sessions with domain experts. However, knowledge transfers in mature domains should be handled differently in order to leverage the existing knowledge. It is a good idea to get into the domain by first reading up on it and then engaging in knowledge-transfers with domain experts.

Eric Evans covers a lot of ground in his talk from the methodology of transferring domain knowledge to diagnosing misalignments between the model and the domain, solutions to correcting such misalignments and strategies for coordinating teams working on the same project. However, I wish he would have talked a bit about the structure and skill-sets of the teams involved in domain-driven development, of the ways to disseminate domain knowledge into a team and of how to engage the domain experts in reviewing and providing feed back on the domain being developed.
All in all a very good presentation which I encourage people to watch.

* The rejection of the analysis phase by agilists could partially be explained by the fact that the agile movement started at a point where appeared brand new domains (such as e-commerce, e-marketing, content management) which were in a continous state of transformation and for which there was no prior knowledge. Analysis in this case was viewed, to a certain extent correctly, as a process which requires significant investements (in order to overcome the lack of knowledge) without a clear return (the results could be obsolete in a few months due to unforeseen changes).
At the same time the revolution in communications created the opportunity to add on stakeholders previously considered un-related to the domain. Encapsulating requirements in an application from unfoereseen stakeholders meant that the domains became more prone to change and harder to predict.
These factors contributed to the misconception that analysis hinders rather than helps.

In this post I will go briefly over some important issues regarding regression testing and they can be used for managing complexity that I have come across in my experience.

A batch of regression tests covering an appropriate part of an application is a very good strategy to deal with growing complexity. Furthermore, the more complex the application the higher the need for regression tests because as systems become more complex the components that make them up tend to become unwieldy (*). In a heterogeneous team with a high variation in skill sets the regression tests are probably best written and carried out by the most experienced members of the team (it goes without saying that the implementation of some functional aspect and the test covering it should be done by 2 separate persons). If the tests are written by the most experienced team members this renders them more expensive, as a result care should be taken regarding the scope of these tests. I generally consider that regression tests should cover higher-value use-cases at a higher scope and that this scope has implications for the testing infrastructure.

The tests themselves should be isolated from testing infrastructure concerns and should focus mostly on the domain that is being covered. A pretty good discussion of these items can be found here. However, the tests should also be run in an environment as close to the production environment as possible, having seen an instance where tests covering an application deployed on a legacy database were passing without even inserting a single record in the DB only to have the application have problems later in PROD because of legacy datastructures.

Manual regression tests are expensive and this high cost has some bad side effects apart from having a higher price-tag: manual regression tests tend to be executed late in the release process. Also, by the fact that they take longer (being manual) they tend to cut deeper into the safety buffer dedicated to fixing bugs found during the release process. In order to solve this problem the regression tests should be in a format which allows them to be carried out early in the development process in order to detect problems early, ideally as close as possible to the development. This would mandate that the regression tests have a high degree of automation and a certain level of embedment into the development process.

Ideally, automated regression tests should be written in a format that ensures that the tests can be manipulated with ease by both developers (in order to be able to run them at an early stage in the development process) and by domain experts/business analysts (in order to prove their validity). A solution which may satisfy these 2 different audiences could be regression tests written in the programming language of the application by experienced team members and which are read-able by domain experts (**). For complex domains the testing infrastructure may need to be adapted in order to satisfy the second point as well.

* Components become out of synch with the required functionality as the complexity that they need to cover grows. Most of the time this misalignment is due the fact that the development pace is too fast and the information for correctly re-designing components for new functionality is unavailable. As components become unwieldy the need to refactoring appears and with it the need for regression tests which are a key component in the refactoring process.

** In practice, however, it is pretty costly to create a test environment that satisfies these 2 different audiences. This is why automated regression tests are typically written in the language used for development with senior team members acting as domain experts for writing tests of validating tests written by team members.