24 May 2010 - 10:03Managing complexity with regression tests

In this post I will go briefly over some important issues regarding regression testing and they can be used for managing complexity that I have come across in my experience.

A batch of regression tests covering an appropriate part of an application is a very good strategy to deal with growing complexity. Furthermore, the more complex the application the higher the need for regression tests because as systems become more complex the components that make them up tend to become unwieldy (*). In a heterogeneous team with a high variation in skill sets the regression tests are probably best written and carried out by the most experienced members of the team (it goes without saying that the implementation of some functional aspect and the test covering it should be done by 2 separate persons). If the tests are written by the most experienced team members this renders them more expensive, as a result care should be taken regarding the scope of these tests. I generally consider that regression tests should cover higher-value use-cases at a higher scope and that this scope has implications for the testing infrastructure.

The tests themselves should be isolated from testing infrastructure concerns and should focus mostly on the domain that is being covered. A pretty good discussion of these items can be found here. However, the tests should also be run in an environment as close to the production environment as possible, having seen an instance where tests covering an application deployed on a legacy database were passing without even inserting a single record in the DB only to have the application have problems later in PROD because of legacy datastructures.

Manual regression tests are expensive and this high cost has some bad side effects apart from having a higher price-tag: manual regression tests tend to be executed late in the release process. Also, by the fact that they take longer (being manual) they tend to cut deeper into the safety buffer dedicated to fixing bugs found during the release process. In order to solve this problem the regression tests should be in a format which allows them to be carried out early in the development process in order to detect problems early, ideally as close as possible to the development. This would mandate that the regression tests have a high degree of automation and a certain level of embedment into the development process.

Ideally, automated regression tests should be written in a format that ensures that the tests can be manipulated with ease by both developers (in order to be able to run them at an early stage in the development process) and by domain experts/business analysts (in order to prove their validity). A solution which may satisfy these 2 different audiences could be regression tests written in the programming language of the application by experienced team members and which are read-able by domain experts (**). For complex domains the testing infrastructure may need to be adapted in order to satisfy the second point as well.

* Components become out of synch with the required functionality as the complexity that they need to cover grows. Most of the time this misalignment is due the fact that the development pace is too fast and the information for correctly re-designing components for new functionality is unavailable. As components become unwieldy the need to refactoring appears and with it the need for regression tests which are a key component in the refactoring process.

** In practice, however, it is pretty costly to create a test environment that satisfies these 2 different audiences. This is why automated regression tests are typically written in the language used for development with senior team members acting as domain experts for writing tests of validating tests written by team members.