24 February 2007 - 20:15SOA - it’s more about service management

I was reading this article on infoQ about service registries. I was struck by the end paragraph:
I periodically get to attend meetings where someone has a vision which involves a single registry/repository to rule them all. There follows a list of all the great metadata which this single truth will “own”. Usually a can’t hear what they’re saying due to the sound of “Darth Vader’s Theme” trumpeting in my lugholes. Empire building.. Empire building ..
This is a pretty fine note about an uber-centralized organization, one that is controlling its entire IT environment from a single command and control center. Which is probably not very efficient because not all services are enterprise-wide and we should not have everything available to everybody (available means available after security checks). Enterprise-wide services - the tax service (*), for example - should be part of this uber-repository, but fine grained services should not. Services particular to a BU or series of BU should not make it into this uber-repository either, but rather in repositories dedicated to them and their partners.
More and more I get the impression that SOA is a lot less about Service Architecture than it is about Service Management. Governance, registries, policy management, etc…, these are simply the means to an end. The current mind-set seems to be that the best way to manage these services is by centralizing them in order to break the IT silos which each BU has created over the years as part of its operations. You need centralized control in order to identify redundant IT functionality and in order to agree on what sub-set of that functionality will be available to all the parties, I agree with this. But I also think that you should recognize the core competencies of each BU and delegate it the ownership of the IT service enabling that core competency. These IT services would be considered BU-specific and should be treated differently from enterprise-wide services (**).

* The tax service: When talking about an enterprise-wide service I usually give the example of a tax service that computes the taxes in a financial institution. Pretty much every transaction executed by all the systems of that financial institution in all the BUs needs to be taxed which raises the need for a taxation enterprise-wide service.

** A BU-specific service may evolve into an enterprise-wide service if the demand for its functionality make it necessary. One important difference between an enterprise-wide service and a BU-specific service is that the enterprise-wide service evolves pretty slowly, every change requiring agreement from a large number of actors.

Later Edit: Service management is essentially a study into the interactions between various various BUs in a large organization. These interactions will then be displayed in the interactions between the BUs’ IT environments.

1 Comment | Tags: Management

20 February 2007 - 16:57Just felt like saying

Who would’ve thought it up someone like you
Just as they brought me round
Now that they brought you down
Roundabout and roundabout
Who wants a life anyway?

Another heart has made the grade
Forget it, forget it, forget it
I don’t understand how the last card is played
But somehow the vital connection is made

No Comments | Tags: Personal

11 February 2007 - 13:41The emergence of the transparent container

I planned to write this post after a series of posts about unusual uses of the Spring framework. I have written so far only one post and I have 2 more started. I don’t have enough time to finish them (I have a small backlog of posts that I started, some 6 months ago) so I thought I just write this one.
My posts are focused on transparent coding: how you would take your current application and apply various business concerns to it transparently without disrupting its normal state. It is a great programming technique which lets you inject new behavior into your application at small costs. It lets you make a small investment and back out of it quickly if this investment turns sour. I look at the transparent container as a binding management tool: it displays you the components of you application and lets you bind one component to another, swap one component for another, bind an advice to a series of methods, etc… It happens that some of the above features are the features of an IoC container but there is much more to this than plain inversion of control.
If I see any problems with this transparent coding (I’ll use this name for lack of a better one) is that there are not that many modelling tools out there because it is a fairly new and unexplored phenomenon. Another problem that I see with it is that the behavior is injected transparently via configuration files. This approach has quite a few management problems, stemming mainly from problems in collaboration (imagine an piece of functionality whose spring.xml files are edited by a few dozen developers). The approach usually taken for this which is to specify these concerns declaratively in a Spring configuration file doesn’t scale as the number of concerns increases, it serves only as a purpose of displaying a new technique. However, Spring could be used as the basis of a platform that would manage all these concerns.
As a sidenote: I see that Spring has got into allowing code for configuration, basically right now you have the option of configuring you application via code rather than simply thru XML descriptors. At the first sight this would simply mean that application configuration has gotten so complex that it requires code for doing it. I think that it also means that the bindings between various components in an application (bindings previously implemented in configuration) have got so complex that they require code for managing them. The configuration file is simply a way to manage these bindings, right now these bindings are becoming overly complex and managing them takes a lot more than configuration. I would say this version of configuring-thru-code (which is actually binding-management thru code) could be where the most interesting Spring development would occur. Code-based configuration would allow for more complex interactions between various components.

I would give this example to display the benefits of code-based “configuration”: let’s say that you have a work-flow engine which models work-flows as a series of tasks, a work-flow engine that currently uses Spring for configuring the work-flows, for defining the flow sequences, etc… You become pretty successful and you sell your engine to a company that uses it for managing dozens of work-flows made up of dozens of tasks.
Now, your customer would like to have some business analysts create new work-flows on the fly (let’s say for simulating a business process), in other words they would need to create, test, discard, record dozens of work-flows looking for, let’s say, the business process with the lowest cost. It is obvious that the configuration file approach (declaring the relationships in spring.xml) is not going to do it because what is required over here is actually implementing a whole work-flow testing process. Now, your work-flow engine revolves around configuring work-flows, so one approach would be to develop or buy an app for testing workflows (any workflow, not necessarily from your application) that would be used by the business analysts looking for the lowest-cost flow and then plug this application into your work-flow configuration via Spring’s new code-based configuration.
The code-based configuration would actually act as a bridge between the work-flow testing system and the work-flow engine itself.
The whole theme of this example is that you are given the option to manage the configuration of your application how you want to (in the above example you wold manage the configuration of your work-flows by piggy-backing on a work-flow testing app), once this configuration cannot be met by the usual spring.xml file. This approach should be taken only when the complexity of the configuration raises to the point where it warrants this investment.

One last point about the transparent container called spring. It has been argued very well that Spring is non-intrusive in the application code. Well, it is intrusive, not in code, but in your application management process. If you are using spring for bindings management you are so dependent on it, you may not be able to continue functioning about it. But about this, maybe some other time.

The transparent container is a container that manages bindings between components and let’s you do it in the manner that you consider best suited to your needs: thru config files, code, etc… This manner is transparent to the component themselves.

Later Edit: I think that managing components relationships in Spring thru code is where the next cool things will occur in the Spring framework.

Later Edit 2: While I have not tried managing complex relationships in Spring thru code I think that this should be an easy process with a low entry-barrier. If this process requires a deep understanding of the Spring framework I’m afraid that it will never be used on the scale it could and it should.

No Comments | Tags: Development, Favorites, Management

10 February 2007 - 3:55Handling large data sets in SQL

For the past week I have been working on an SQL script that was updating a few dozens millions of records in a table. It was very interesting and it was my first occasion at handling data sets so large. I first suggested Tom Kyte’s famous approach to updating very large tables, but given the fact that the team never tried something like this and that the table was so large they decided to play safe and do the job with regular update statements.
What did I learn while working on this?
1) Update in small chunks otherwise your logs will blow-up your partition. You have to find a way to split your data in chunks that you know will not explode your rollback segments.
2) Use indexes very carefully!!!! In my case I needed to build some hash temporary tables in order to hit some indexes, and in order to build those tables efficiently I needed to figure out how to hit other indexes. The path carved thru the data was pretty interesting…
3) Always determine the execution plan before carrying out the query in order to see what indexes you are hitting. A ’set showplan on, set noexec on’ should do the job on Sybase.
4) Know your data. Among other things it will help you find the appropriate indexes. It will also help you choose between different queries, check out this for an example of 2 different queries which return the same result but are tuned for different data sets.
5) This is not really related to SQL, but I thought I should say it: Don’t bother to optimize your data structures in Java unless you really have to. Don’t spend hours debating whether you should use a TreeMap vs a HashMap for a Map that will hold at best a few hundred items, do this when you are handling large data sets. The reason for this is that optimizing a data structure for access is so dependent on data (on how it is dispersed, on what you are querying, on how this data will grow, etc…) that you don’t have enough input for a small data set.
6) Efficient data access is tied to the data itself very strongly.
7) Prepare for this. I have been reading for the past months asKTom, Jonathan Lewis’s blog and Tom Kyte’s blog. These are good resources for starting.

Gotta go!!

Later Edit: While reading this post I started thinking about the duality of data vs functionality. Most developers spend most of their time dealing with functionality, I think they are missing out on a few interesting things. Another thought would be that if you have a problem regarding data the best person to talk about it would probably be a DBA, it probably has seen data more data than you can handle in a wide variety of formats. Just a thought.

No Comments | Tags: Development

6 February 2007 - 19:08Language re-use

I was reading this article about some guy implementing some JSR that I don’t really care about. What I really liked about this article is this quote:
Until java has first class support for properties, some syntax will need to be adopted. EL was created for needs similar to beans binding and is a natural fit. No point in inventing a new duplicate syntax.
You are probably familiar with the concept of re-using classes, packages, frameworks, etc…, but are you familiar with the concept of re-using languages? Pretty neat…
This reminds me of another guy who worked on a natural language processor: he had to ‘integrate’ some linguists into the application, basically the linguists had to create some rules according to which text was supposed to be parsed. The linguists needed a very flexible environment for expressing those rules, something very similar to a computer language. So my friend looked around and decided that the linguists will learn regular expressions and create their rules in regular expressions. He basically re-used a language too. Pretty neat again.

No Comments | Tags: Development

4 February 2007 - 21:13Customizing applications transparently thru interceptions

In this post I will be considering applying business logic transparently thru interception for some specific scenarios and discuss the pros and cons of such an approach.
Let’s say that you have a content-management system (CMS) and a forum application, the CMS being written in Java. The CMS is used for managing an online paper and the forum is used for managing a community of users interested in your content.
This is the original CMS model:
public interface PublishingBean{

// this is the method that publishes an item
public void publishItem(Item item) throws PublishingException;

}
public class DefaultPublishingBean implements PublishingBean{

public void publishItem(Item item) throws PublishingException{

// format the item
formatItem(item);// save it hard-disk
saveItem(item);

// syndicate it
populateRSSFeed(item);

}

// some formatting: adding menus, putting up banners, etc…
protected void formatItem(Item) throws PublishingException{
// format the item
}

}
DefaultPublishingBean is registered in Spring to implement the publishing bean:
<bean id=”publishingBean” class=”DefaultPublishingBean”/>

Now let’s say that you want to integrate the CMS with the forum. The integration will be very simple, at the end of each item you may or may not put a link which would create or update a thread on the forum related to that item. Part of this integration is a new publishing bean which contains logic interacting with the forum:
public class ForumPublishingBean implements PublishingBean extends DefaultPublishingBean{

// this method prepares the bean so that it doesn’t display the link to the forum
public void donotDisplayForumLink (boolean bool){
// set a flag
}// this method prepares the bean so that it links to the forum in ‘moderated’ mode
public void displayModeratedForumLink(boolean bool){
// set a flag
}

protected void formatItem(Item) throws PublishingException{
// if ‘do not display forum link’ mode is on then do not display the item

// if ‘display moderated forum link’ mode is on then put-up then display a moderated link

// else display the link at the bottom of the item.
}

}
ForumPublishingBean just needs to re-format the item, it delegates saving to disk, syndication, etc… to the super-class because this functionality didn’t change.
Right now the ForumPublishingBean replaces DefaultPublishingBean for publishing beans by the means of this bean definition:
<bean id=”publishingBean” class=”ForumPublishingBean”/>

OK, this was the set-up, now we are ready to start on the topic of this post. Everything is running OK, editors are happy,the community keeps talking on the forum when, all of a sudden, you hire Ann Coulter and Charles Kas an editor ;-), both in the ‘Politics’ section of your newspaper.
The result of hiring Ann is that a flame war starts raging on the forum any time Ann posts an article. The business starts getting fidgety and decide that Ann’s articles should not have the link towards the forum, this would limit the flames. This is business requirement #1: that Ann’s, and only Ann’s, articles should not provide a link towards the forum.
Next, Charles is not the flamethrower that Ann is, but he is not small potatoes either, as a result a lot of his items are getting flamed on the forum. The business decides that Charles’s articles should link to the forum in ‘moderated’ mode. This is business requirement #2.
Next, let’s leave politics aside for a while ;-), Vista is about to launch and the technology section of the paper is getting flamed on the forum by both Linux and MS die-hards. The business decides that for a while every article containing the words Vista should have moderated links to the forum and it doesn’t know when this rule will expire. This is business requirement #3.

So you have 3 requirements, quite different one from the other, very limited in scope and which should not pollute in any way either the CMS or the forum’s code base. Moreover, you are not sure in what direction these requirements will evolve, they are pretty specific. Do these requirements actually mean that you need to integrate the authors into the forum-CMS integration? Or do they mean that some sections, ‘Politics’ in this case, need moderation? It is pretty early to draw any assumptions from these business requirements because they are very poor.
One way to implement these business requirements without making any broader assumptions is to intercept the calls to the Publishing Bean and inject this very specific business logic. This could be done with 3 around advices as shown below:

import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
// this interceptor will check if the author of the Item that is getting published
// should have its forum link removed and remove its forum link
public class class BlockForumLinkByAuthorInterceptor implements MethodInterceptor{

private List blockedAuthors = new ArrayList();
public void setBlockedAuthors(List blockedAuthors){..}public Object invoke(MethodInvocation invocation){

// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
if(blockedAuthors.contains(cmsItem.getAuthor())){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// block this author’s forum link
publishBean.donotDisplayForumLink(true);// publish the item
publishBean.publish(cmsItem);

}else{

// go ahead with the original method
invocation.proceed();

}

}

}

// this interceptor will check if the author of the Item that is getting published
// should have its forum link moderated and moderate it
public class class ModerateForumLinkByAuthorInterceptor implements MethodInterceptor{

private List moderatedAuthors = new ArrayList();
public void setModeratedAuthors(List authors){..}public Object invoke(MethodInvocation invocation){

// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
if(moderatedAuthors.contains(cmsItem.getAuthor())){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// moderate this author’s forum link
publishBean.displayModeratedForumLink(true);// publish the item
publishBean.publish(cmsItem);

}else{

// go ahead with the original method
invocation.proceed();

}

}

}

// this interceptor will check if the section of the Item that is getting published
// should have its forum link moderated and moderate its forum link
public class class ModerateForumLinkByKeywordsInterceptor implements MethodInterceptor{

private List moderatedKeywords = new ArrayList();
public void setModeratedKeywords(List moderatedKeywords){..}public Object invoke(MethodInvocation invocation){

boolean moderateFlag = false;
// get the cmsItem that is getting published
Item cmsItem = (Item)invocation.getArguments()[0];
foreach(moderatedKeyword in moderatedKeywords){

if(cmsItem.getText().indexOf(moderatedKeyword) != -1){
moderateFlag = true;
}

}
if(moderateFlag){

ForumPublishingBean publishBean = (ForumPublishingBean)invocation.getThis();
// moderate this story’s forum link
publishBean.displayModeratedForumLink(true);

// publish the item
publishBean.publish(cmsItem);
}else{

// go ahead with the original method
invocation.proceed();

}

}

}
The above examples are woven into the application via this XML configuration:
   <!– The previous bean becomes the target of the advices above –>
   <bean id=”publishingBeanTarget” class=”ForumPublishingBean”/>
   
   <!– Register the 3 advices –>
   <bean id=”blockForumLinkByAuthorInterceptor” class=”BlockForumLinkByAuthorInterceptor”>
      <property name=”blockedAuthors” value=”Ann Coulter”/>
   </bean>
   <bean id=”moderateForumLinkByAuthorInterceptor” class=”ModerateForumLinkByAuthorInterceptor”>
      <property name=”moderatedAuthors” value=”Charles Krauthammer”/>
   </bean>
   <bean id=”moderateForumLinkByKeywordsInterceptor” class=”ModerateForumLinkByKeywordsInterceptor”>
      <property name=”moderatedKeywords” value=”Vista”/>
   </bean>
   
   <!– Set-up the proxy which replaces the original ForumPublishingBean –>
   <bean id=”publishingBean” class=”com,springframework.aop.framework.ProxyFactoryBean”>
      <property name=”proxyInterfaces” value=”ForumPublishingBean”/>
      <property name=”interceptorNames”>
         <list>
            <value>blockForumLinkByAuthorInterceptor</value>
            <value>moderateForumLinkByAuthorInterceptor</value>
            <value>moderateForumLinkByKeywordsInterceptor</value>
         </list>
      </property>
      <property name=”target”>
         <!– point this proxy to the original ForumPublishingBean –>
         <ref=”publishingBeanTarget”/>
      </property>
   </bean>

And this is pretty much it. The code above was using around advices, depending on the use-case you could have used other types of advices. Well, so we have implemented these very specific business rules transparently and without any intrusion into the application’s original code, which is not too bad. Now comes the hard part: when should you take this approach? (You know, the hard question is not how, but rather why ;-))

I think this approach should be taken when you are given some special business rules which you cannot generalize. In the above example it was pretty hard to determine what was driving the editing of the forum links: the author’s type, the words used in the CMS Item, etc… In a sense this approach buys you time during which you can watch the demand for a particular business requirement grow or stagnate and then decide whether to make an investement in handling it or not. In our example the interceptors that moderates forum links by author lets you see if this behavior is needed before modeling and implementing it: if you have 20 authors that need to be moderated then it probably pays off to model the moderation process, if you have only one it doesn’t make sense to invest time in it.

You could also take this approach when you don’t see the benefit of setting up a whole infrastructure in order to implement this business rules, when for example it doesn’t make sense to change the user management module in order to change the behavior of 2 or 3 users. In my experience I had a few cases where I needed to change a whole user module because one user was not supposed to click on a button, I wish I could have simply intercepted his calls and inject this very specific behavior.

You could also use this approach if you need to implement a simple business requirement that is set to expire soon. In our case we decided to moderate every CMS item that contains the word Vista in it. This business rule will probably be dropped in a few months once the excitement over Vista wears off. It doesn’t make sense to make a significant investment of code into something that will expire so soon, if you can get by with a small hack you may consider it. The beauty of this example is that you can shed this hack very easily when the need for it expires .

You should note that this example doesn’t scale very well and that it uses values hard-coded in configuration files. If this causes problems in your application, then probably you should not follow this example. Getting into scale problems is actually an indication of the fact that this interception framework is not servicing your needs anymore and that you should probably do something different about implementing this requirement.

One potential source of problems is the mushrooming of these business-related interceptors. Mushrooming code usually points to bad design, mushrooming interceptors points pretty much to the same thing. You should watch out if these interceptors are getting out of whack and once a certain threshold is met find a way to consolidate them, probably by making some generalizations about their roles.

One interesting thing about these interceptors is that they are not cross-cutting concerns, rather they are very specific about their target. I initially thought about using the title ‘Customizing applications transparently thru AOP’ but then I realized that I am not talking about aspects (which are cross-cutting concerns), but about very specific joinpoints.
Another interesting thing is that these are business-related interceptors and not infrastructure interceptors.

Well, this concludes this post. I wanted to have more time to talk about this, but I do not have the time. Feel free to drop some comments, I find this type of development pretty interesting.

P.S. I chose Ann Coulter and Charles Krauthammer because I needed an example of a business concern very limited in scope yet very important to the business. I didn’t have time to cook-up a better example so I just thought this one out. It is a bit off the wall, but again, I simply didn’t have the time. I chose some controversial authors at random in order to set-up a very specific business rule.

Later Edit As far as I am concerned I think the this approach is just another way of getting your development cycle closer to the business needs. This approach avoids building a whole infrastructure for very small business requirements and rather it lets you watch these business requirements grow before making an IT investment in them.

No Comments | Tags: Development, Favorites, Management