12 August 2008 - 18:38Relaxing ACID properties as you scale out

A while ago I looked at the transition currently occuring in enterprise computing from ACID transactions to compensation by message passing. I concluded by saying that relaxing ACID properties and achieving state alignment via message passing is essentially shifting the costs from the costs of ACID transactions (such as database rows locking) to complexity costs (such as managing the growing number of messages between systems which are compensating for these relaxed transactions). Another side effect is the introduction of latency in this state alignment process: thru ACID transactions state alignment is achieved at the time the operation takes place while thru message passing it occurs some time later.

I was watching this interview on infoq and I found that Gregor Hohpe pretty much voiced the same opinions, that as you scale out it is impossible to achieve state alignment at the time the operation and that you need to compensate for it by passing messages.These complexity costs are essentially the costs of managing business logic as it spawns across multiple systems, applications, departments, etc… State alignment in these cases consists primarily of having various business analysts agree on how these systems should interact (should this trade go to this depot for that security for that client) and then implementing the interaction between these systems. One thing that is not very obvious is that the cost of interaction between systems is pretty close to the cost of interaction between the departments running these systems and that as you scale out you need to be able to accomodate or change various departments you are encountering (*).

It is interesting to watch the interview from this point of view because you get to see the costly hacks that various organizations got to make in order to get their systems to talk to one another (the horrific “magic account” used in one bank makes one hair stand up). Gregor goes as far as saying that these ways to deal with these problems are pretty common, so common actually that at one point he gets defensive about the sloppiness which comes thru these examples. The fact that these problems are wide-spread is a pretty bad excuse, ideally these systems would be architected so that we do not need to add one layer of hacks on top of another in order to get something of value out of them.

At the end of this interview I ended up with the impression that we are on the first iteration of relaxing the ACID properties and that I got a pretty good glimpse into the costs associated with this relaxation. Sometime in the future patterns for relaxing these properties will appear and problems common in all these compensations will be solved. I think that these patterns will be more oriented towards business-logic and management rather than towards technology. Efficient interactions revolve around small transaction costs and minimizing transaction costs will be part of the solution of managing this growing complexity.

* Interestingly enough an SOA strategy will probably take these costs down if it manages to impose procedures so that various departments interact one with another more efficiently. However, I am pretty skeptical that we will see a lot of progress soon on this front.