Sunday, August 27, 2006

Coverage Generation or Generations of Coverage?

While I was thinking of this post, I came along the "32 & 16 Years Ago" section of the July 2006 issue of Computer and was amazed by what I read in the entry titled "Software Testing". It started like this: "The objective of (...) probe insertion mechanisms (...) is not to find errors; rather it is to quantitatively assess how thoroughly a program has been exercised by a set of test cases" (edited by Computer).

This is dated... July 1976! Code coverage in 1976! Golly! What a blunt reminder that we are, at most, building on the shoulders of giants... Humility is definitively the only path that is safe to walk.

Anyway, my current concern with code coverage was to fix the 100% target that I was talking about in my "Programming for Agility" post and that I borrowed from Uncle Bob himself. This is a nice high level target but can it really be reached? I do not have much doubts for code I am writing with my little busy hands but what with generated code? Especially code that contains many conditional branches like the one Eclipse generates for equals() and hashCode() methods?

public boolean equals(Object obj) {
if (this == obj)
return true;

if (obj == null)
return false;

if (getClass() != obj.getClass())
return false;

final DefaultTodoListItem other = (DefaultTodoListItem) obj;

if (id !=
return false;

return true;
I came with three possible strategies, including one for which I already apologize:
  1. Exclude generated code from test coverage estimation: for having written this one, I have already slapped my hand as a due punishment* ! Generated code does not mean intrinsically safe code: it is live code and is subjected to all sorts of possible errors you can imagine.

  2. Use test code generators: great tools like Agitar Agitator can very conveniently generate test cases for all sorts of extreme values that will most probably explore all the possible execution paths of the aforementioned generated methods. These tests would complement nicely the ones written by the developers.

  3. Reduce the number of needed tests by enforcing class invariants: this can be done by working with immutable objects and check for invalid or exceptional conditions at construction time. This also allows to pre-calculate values like hashCode() and toString() rendering. The tests will then target user written code and will exercise construction of invalid objects instead of testing the calls of all methods possibly sensitive to an invalid state object.
Please do share other possible approaches or practices you think of or daily use in this matter. Even if we are the second generation of code coverage buffs, we might still have something useful to say!

* An efficient self-punishment technique I learnt from J.B. Rainsberger during SD West 2005.