Infected but not driven

The least I can say is that I'm test infected: when a coverage report shows lines of code that are not exercised by any test, I can't help but freak out a little (unless it appears that this code is truly useless and can be mercilessly pruned). This quasi obsession for testing is not vain at all: time and again I have experienced the quality, stability and freedom of move a high test coverage gives me. Things work, regressions are rare and refactoring is a bliss thanks to the safety net tests provide.

So what's with TDD... and me?

There are some interesting discussions going on around TDD and its applicability, which I think are mostly fueled by the heavy insistence of TDD advocates on their particular way of approaching software development in general and testing in particular. The more time I spend thinking about these discussions, the more it becomes clear to me that as far as testing is concerned, the usual rule of precaution of our industry applies: ie. it depends.

To be frank, I'm having a hard time with the middle D in TDD: as I said, I'm test infected, low test coverage gives me the creeps, but my process of building software is not driven by tests. From an external viewpoint, it is driven by features so that would make it FDD. From my personal viewpoint, it is driven by gratification, which makes it GDD.

Being gratified when writing software is what has driven me since I'm a kid: I didn't spend countless hours hurting my fingers on a flat and painful ZX-81 keyboard for the sake of it. I did it to see my programs turned into tangible actions on the computer. It was gratifying. And this is what I'm still looking for when writing software.

But let's go back to the main point of this discussion: TDD. With all the industry notables heavy-weighting on writing code while being driven by test, should I conclude there's something wrong with my practice? Or is the insistence on test first just a way to have developers write tests at all?

Adding features to a system, at least for the kind of systems I'm working on, mainly consist in implementing a behavior and exposing it through some sort of a public interface. Let's consider these two activities and how testing relates to them.

Behavior

When I write simple utility functions, like chewing on some binary or data structure and spitting out a result, I will certainly write tests first because I will be able to express the complete intended behavior of the function with these tests.

Unfortunately, most of the functions I write are not that trivial: they interact with functions in other modules in non-obvious manners (asynchronously) and support different failure scenarios. Following a common Erlang idiom, these functions often end up replying a simple ok: such a result is not enough to drive the development of the function (else fun() -> ok end would be the only function to write to be done). In fact, testing first this kind of functions implies expressing with mock expectations all the interactions that will happen when calling the top function. That's MDD (Mock Driven Development) and it's only a letter away from making me MAD. Sorry but writing mocks first makes me nauseous.

My approach to developing and testing complex functions is, to me, more palatable as it leads to a faster gratification: I start by creating an empty function. Then I fill it with a blueprint of the main interactions I am envisionning expressed as comments. Afterwards, I reify this blueprint by turning the few comments in the original function into a cascade of smaller functions. At this point, I fire-up the application and manually exercise the new function: this is when the fun begins as I see this new code coming to life, finding implementation bugs and fixing potential oversights. After being gratified with running code, I then proceed to unit test it thoroughly, exploring each failure scenarios with mocks and using a code coverage tool to ensure I haven't forgotten any execution branch in my tests.

This said, there is another behavior-related circumstance under which I will write tests first: when the implemented behavior is proven wrong. In that case, writing tests that make the problem visible before fixing it is the best approach to debugging as it deals with the problem of bad days and lurking regressions.

Public Interface

Writing usable modules imply designing interfaces that are convenient to use. Discussing good API design is way beyond the point here. The point is: could writing tests first be a good guide for creating good interfaces? The immediate answer is yes, as by eating your own dog food first makes you more inclined into cooking it into the best palatable form possible (anyone who has had to eat dog food, say while enduring hazing, knows this is a parabola).

In my practice, I have found things to be a little different, again for less than trivial functions, which unfortunately compose most of a complex production system. For this class of functions, I have found that the context of a unit test is seldom enough to fairly represent the actual context where the functions will be used. And consequently, the capacity to infer a well-designed interface based on these tests first and alone is not enough. Indeed, a unit test context is not reality: look at all the mocks in it, don't they make the whole set look like a movie stage? Do you think it's air you're breathing?

When creating non-trivial public functions, I've found a great help into going through a serious amount of code reading in the different places where it is envisioned these functions will be used. Reading a lot of code before writing a little of it is commonplace in our industry: while going through the reading phase, you're actually loading all sort of contextual information in your short term memory. Armed with such a mental model, it becomes possible to design new moving parts that will naturally fit in this edifice. So that I guess that practice would be RDD (reading driven development).

Daring to conclude?

I find it hard to conclude anything from the dichotomy between my practices as opposed to what TDD proponents advocate. I consider myself a well-rounded software professional producing code of reasonably good quality: unless I'm completely misguided about myself, I think the conclusion is that it's possible to write solid production code without doing it in a test-driven fashion. If you have the discipline to write tests, you can afford to not being driven by them.

Is Test Overlap A Necessary Evil?

In a recent blog post titled "The Limitations of TDD", Jolt Awards colleague Andrew Binstock shared some reservations Cédric Beust has about TDD. When a person of extensive experience like Cédric speaks about testing, you pay attention. And I did.

Among the very interesting quotes from Cédric that Andrew has reproduced, the following really struck me:

Another important point is that unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools.

You also bring up another interesting point: overtesting can lead to paralysis. I can imagine reaching a point where you don't want to modify your code because you will have too many tests to update (especially in dynamically typed languages, where you can't use tools that will automate this refactoring for you). The lesson here is to do your best so that your tests don't overlap.

Trust me, as a test-infected developer, I would love to stay in a state of self-delusion and pretend that test-induced paralysis doesn't exist. But that would be a lie: the reality is grimmer than the wonderland of testing I would wish to live in. The reality is that tests both encourage and resist change.

On the one hand, tests encourage and support refactoring: when the behavior of the application should not change but the code needs to be re-organized, tests are a blessing. They give you the courage to dare changing code because of the immediate feedback they give when you've been refactoring a little too aggressively. And this is priceless.

On the other hand, tests resist behavioral changes. Because tests have captured all the nitty-gritty of your application, when comes the time to change its behavior, you will need to invest time to adapt your tests accordingly, and this whether you rework the tests first or not. As Cédric pointed out, in a dynamically typed language, this is immensely painful as development tools are almost useless in assisting you with the required changes. Similarly, if you use mock objects, you are good for going down a deeper Circle of Hell, where more painful and frustrating manual fixes await you.

So, is there any hope out of this love / hate relationship? Knowing that "the only way to go fast is to go well" dumping tests altogether is certainly not an option. Could the solution lies in Cédric's very last words: "do your best so that your tests don't overlap"?

At this point, I don't know yet but I've decided that, as a starting point, I should start to estimate the amount of overlap I'm dealing with in the Erlang game server I'm working on. Interestingly, what I've found could pretty much apply to the vast majority of Java projects I've been previously working on. Maybe it applies to your projects too?

The first thing I've looked at is the testing overlap that exists between two layers of our application:

As you can see, the overlap exists because tests of the upper layer rely on mocks to simulate all the happy paths and most of the unhappy paths of the underlying layer. The overlap is not total because a layer tend to reduce the granularity of the unhappy paths it faces internally in order to expose the upper layer to a limited amount of bad situations to deal with. Hence the limited amount of mocked features in the overlap area.

When applied to a typical vertical slice of our system, it looks like this:

This is not too bad. Until the wind of feature change comes blowing on this mock-based card-house of tests, life is peachy.

Until now, the tests I have been looking at were only unit and database ones. If I add our functional tests on top of the overlap diagram, here is what I get:

Now the application container is also tested, plus we get an insane amount of overlap.

But the amount of overlap is not what I want to discuss first: it's the test coverage profile that I want to look at first. Notice how the functional tests explore less unhappy paths as they exercise deeper application layers. This can be explained simply: some unhappy paths are very hard to reproduce via the reduced set of functionalities exposed at the top level, oftentimes because they require a very specific and complex state to be established beforehand or conditions that could only be met in case of low level failures (loss of networking, for example).

It's obviously out of the question to consider dropping functional tests in order to reduce the testing overlap. As Cédric said, they are the only tests that have a true value for the end user of the system. My experience confirms that you can reach a nearly flawless first-time client integration if your functional tests have a coverage profile that is similar to the one in the last figure above.

The only problem lies in the quality of feedback you get from functional testing: because it's impossible to make the gory details of the errors encountered when exploring unhappy paths surface at the uppermost level, your system must have a solid logging strategy that allows you to precisely track issues, should you decide to code using functional tests as your only safety net.

So are the unit tests overlapped by the functional tests the ones that must go? Cédric again gives the answer: if time is short, it's better to focus on the functional tests. Of course, if you have a battery of unit tests in place, keep them.

But, maybe, just maybe, as you move to your next project, consider writing functional tests firsts? That way you would have built first the tests that truly matter and, if time permits, write unit tests as you implement the features expected by the functional tests.

Software Manifestos: A Matter Of Trust?

As software manifestos have started to proliferate these past months, I have started to wonder what could be the root cause for their creation. Why would thought leaders gather, assert a small set of values and shrink-wrap them as a manifesto, calling for others to sign it? My feeling is that these manifestos are the expression of a pushback on a particular aspect of software development that went insane.

Here is a little game: match the manifestos with the software insanities they push back on:

Big methodology and design up-front		Software craftsmanship manifesto
Army of flying monkeys testing		Agile manifesto
Snake-oil vendors and ivory tower architecture		QA manifesto
Reckless programmers and incompetent coders		SOA manifesto

(One manifesto I see missing here is the "recruiter manifesto", which should push back on inane keyword-driven head hunting schemes solely able to put the wrong people at the wrong spots)

If we dig deeper, we become tempted to ask why is our industry suffering from such insanities? What does make software different? Could it be because of complexity?

Complexity. Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine--open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound.

Frederick P. Brooks, Jr., No Silver Bullet

The natural reaction to complexity is to try to escape it at all cost, even if it means wilfully practising self-deception. Hence silver bullets, hence snake oil vendors, hence all these methodologies, governance committees and ivory towers that are there to nurse the insecurity of higher levels of management by giving them the impression software creation is under control and, finally, out of the hands of programmers.

Of course, it doesn't work that way: years and millions of dollars later, reality comes knocking at the door, manifestos are getting written and everyone is sent back to the same fundamental question they've been trying so hard to avoid: how to build trust in software developers?

And that's of course a question for us, software developers. How can we build such a trust in us when so many forces are pushing towards the opposite?

Granted that software development is unpredictably complex and that this complexity reveals itself when the devil shows up (those pesky details), it is clear that the overall battle of trust is fought during each decision, when tackling each detail and writing each line of code.

I think we could learn a few lessons from the world of aviation, where trust in pilots has been built progressively and methodically. When you fly an airplane, you have plenty of decisions to make and losing any of these battles can end up very badly for everyone. So why are pilots trusted? Aren't they fully superseded by ATC anyway? Answer is no: even if ATC has authority, the PIC (Pilot In Command) has the last word because he is the one out there dealing with the ultimate reality of flight. Despite its authority, ATC doesn't micro manage the pilot: the pilot is in-command.

To have the privilege to be a PIC, you have to remain current and regularly prove that you can be trusted for your judgement based on your skills, experience and training.

If the acronyms didn't sound so bad, I would dare suggesting programmers should become DICs, ie Developers In Command. Though working under different forms of authority, DICs would be fully trusted for taking the final decisions in the daily battle of writing code. In this world, it wouldn't be an heresy to say that developers could build large and complex software systems from the ground up, without the need for snake oil, committees or big design.

When trust will be manifested, we won't need manifestos anymore.

Looking for my seams

Like any test infected programmer switching to a new development platform, I have spent my first days working with Erlang looking for my seams. Here, I am talking about seams as defined by Michael Feathers in Working Effectively with Legacy Code: "A seam is a place where you can alter behavior in your program without editing in that place." As such, seams are key enablers for unit testing as they allow you to redirect calls leading out of your SUT to mocks or stubs or any kind of test double you tend to favor.

In object oriented programming, this is a given thanks to polymorphism and dependency injection. But in Erlang, where SUTs are MUTs (modules under test) and the common idiom for invoking a method is module:function(parameters), things are a little less obvious. Indeed, hard-wired function calls from one module to another don't leave much room for any kind of substitution. Without the capacity to fully test my modules in independence, I quickly started to feel uneasy. After a few days, it felt like free-falling without a parachute.

Then I started to seriously investigate my options...


Macros

Macros allow you to define blocks of instructions that the pre-compiler will substitute for you at the different places you refer them. When used in conjunction with flow control statements, macros can be used to switch one code fragment with another one by passing a parameter to the compiler. This seems to fit the bill as you can use conditional macros to alter behavior without editing the places where the macro is used.

This said, I have quickly ruled out the use of macros as a valid seam. Imagine having to do this for all the function calls leading out of the MUT:
<br />-ifdef(unittest).<br />-define(CALL_FOO(X), foo_test:function(X)).<br />-else.<br />-define(CALL_FOO(X), foo:function(X)).<br />-endif.<br />
Moreover, if a mistake exists in the non-unit test wiring part of the conditional macro, I would have had to wait for integration tests or actual deployment to get feedback on the issue.


Funs

Though the common idiom is to early bind the module and function you want to call, Erlang is fully capable of late binding and dynamic invocation, as this very crude example illustrate:
<br />1> Square = fun(X) -> X * X end.<br />#Fun<erl_eval.6.13229925><br />2> Square(3). <br />9<br />
This opens interesting possibilities for MUTs that expose higher order functions. If the function that must be tested accepts one or several functions, passing a mock implementation is just a matter of providing an anonymous function of the same arity. This mock would perform nothing besides storing the received parameters in a shared storage, like the process dictionary, for later inspection.

Unfortunately, not all functions receive their dependencies as parameters but instead perform direct calls to other functions in other modules. It could be a plausible and drastic design decision to forbid all direct inter-module calls in favor of passing dependencies as anonymous functions via additional arguments. Some have suggested to use a record to pass around all your application dependencies as a single extra argument added to all functions.

Interesting but the idea of polluting all functions with additional arguments is less than palatable. In fact, it would great if these extra arguments could be defined module-wise and implicitly added to each of its functions... Rejoice! Parameterized modules have been introduced to perform exactly this delicious syntactic sugar trick!


Parameterized modules

I have discovered parameterized modules while writing controllers for Mochiweb. In this pretty cool HTTP server, the request reference that your processing function receives points to a parameterized module, allowing this kind of neat syntax:
<br />Method = Request.get(method).<br />
Though this may feel object oriented, don't get fooled: behind the scene, there is no instance of anything. The Request reference contains all the hidden parameters that the get function needs besides the atom specifying what you want to get. Behind the scene, what really happens is more likely something like that:
<br />Method = Request.get(method, Socket, Method, RawPath, Version, Headers).<br />
But because the Mochiweb Request is a parameterized module, all the extra parameters have been specified once, packed in the reference and stay hidden there for your utmost convenience!

From there, it's easy to see how to write stubs for parameterized modules: just write another parameterized module that export functions with the same signature as the ones you use in the real module. Here is a very incomplete but fully working request stub for Mochiweb:
<br />-module(mochiweb_request_stub, [Method, Path, Headers, Parameters]).<br /><br />-export([get/1, parse_post/0, respond/1, not_found/0]).<br /><br />-define(SAVE_RESPONSE, mochiweb_request_stub_response).<br /><br />get(method) -> Method;<br />get(path) -> Path;<br />get(headers) -> Headers;<br />get(response) -> erlang:get(?SAVE_RESPONSE);<br />get(path_tokens) -> string:tokens(Path, "/").<br /><br />parse_post() -> Parameters.<br /><br />respond(Response) -> put(?SAVE_RESPONSE, Response).<br /><br />not_found() -> put(?SAVE_RESPONSE, not_found).<br />
Note how I use the process dictionary to store values that I will later retrieve for asserting everything went as expected. By using parameterized modules, I have been able to reach near 100% code coverage. Does this mean parameterized modules are the best thing since sliced bread?

Well, so much for the free lunch as there are some drawbacks to consider:

• Experimental - Parameterized modules are still officially considered as an experimental feature of Erlang, hence subject to change. Unlike the Java world where everything is kept for ever just in case, Erlang doesn't patronize developers, so if this feature is one day bound to oblivion, it will be tossed out. And quickly.
• Unchecked - Unlike with a direct module's function reference, compile-time checking is not available, leading to possible bad surprises at runtime. If the parameterized module reference your code uses does not expose the expected function, you're in for a nasty error. In fact, you can totally pass a reference to a Foo module while your function expects a totally unrelated Bar module. As a tentative mitigation, I have added a verification function in my modules so they ensure at start-up time they are correctly wired. This feels like framework-envy,so I'm not fully satisfied with this approach.
  
• Confusing - Because the actual module is not directly referred to, reading such code becomes more complicated. You have to infer from the context (or some coding conventions, or even comments) what is the module that will actually be wired-in at runtime. Decreasing understandability is definitively not a good thing.
  

Besides these downsides, I still believe that the complete MUT isolation and behavior swapping facilities offered by parameterized modules make them a very interesting tool for the test-minded Erlang developer.


Closing notes

MUTs have other kinds of dependencies that you will want to substitute at unit testing time. To name a few:

• Process dependencies - A MUT can contain functions that directly depend on other processes via their PIDs (process IDs). An interesting seam here is the local registry of processes (and ports) that you can use to set-up test processes and register them under the same name as the ones used at runtime.
  
• Mnesia - Stubbing out calls from the controllers to the DAO is a good strategy but what about the DAO itself? Instead of stubbing out each Mnesia call, I have opted for running it in-memory at unit test time (à la hsqlddb) and activating file persistence only at runtime. This is extremely fast so very well suited for the task.

Finally, if you wonder what unit testing framework I am using, I have opted for etap, which I find very simple and powerful enough for my needs. If you want something more structured and feature-rich, EUnit is the answer.

Free fall is over: I have found my seams and landed seamlessly. Please share your own test infected adventures in Erlang.


UPDATE 23-SEP-2009: Hot code swapping is also a very powerful seam, that has been smartly leveraged to create ErlyMock, a quite capable mock framework for Erlang.

Strict Unit Tests for Public Data Contracts

Suppose we have the following code:
public class Thinger {<br /> private static final RESULT = "Joy";<br /><br /> public String doThing() {<br /> return RESULT;<br /> }<br />}
When testing such a code, it is tempting to modify the visibility of RESULT to package protected in order to write tests that share the constant value:
<br />@Test<br />public void doThingWorks() {<br /> assertEquals(Thinger.RESULT, thinger.doThing());<br />}
After all, reuse is good, right?

Well, in this case, I think that reusing this constant is not a good idea if your API is a public one (or if this code gets exposed as a service, which is practically the same thing).

In fact, I advocate to write the test like this:
<br />@Test<br />public void doThingWorks() {<br /> assertEquals("Joy", thinger.doThing());<br />}
But why the duplication?

The catch with public APIs is that they create long lasting expectations in an uncontrolled number of users and systems. Consequently, stability should be their essential characteristic. Through interfaces, it is easy to provide an illusion of stability: as long as the API is backwards-compatible binary-wise (or operation-wise for services), it can change at will and life is peachy.

So what is so risky with the static field above?

Well, the fact of the matter is that the value returned by the doThing() method is also part of the contract. Indeed, beyond the object-oriented concept of interface, data is also part of the overall contract with a particular class (or service). So data should exhibit the same stability as the interface itself.

When sharing a constant in the unit test, it is possible to modify the data contract without noticing. Suppose I change the value of RESULT from "Joy" to "Happy". The first test will give me a green light, while the second will be red. And it is the latter that I am looking for: I want my unit tests to tell me that I have broken the data contract of my class.

Not its users...

Jitter Mule Functional Tests With Jitr

Embedding Mule in a web application allows you to tap the Servlet layer of your favorite web container, which is a good thing as you are supposedly very familiar with its behavior and tuning.

When it comes to writing functional tests for such an application, my strategy was to replace the Servlet endpoints with stock HTTP ones, a substitution that is trivial to perform thanks to the modularity of Mule configuration files: I simply loaded a slightly different set of files at functional test time than the actual file configured in my web.xml file. Since I was writing the tests as subclasses of org.mule.tck.FunctionalTestCase, all the goodness of Mule was readily available as protected methods or members.

Still, I was not very satisfied with this approach because of the discrepancy between what my functional tests were exercising (the stock HTTP transport) and what I was actually using (the Servlet transport). This discrepancy bit me badly in the past weeks with a problem only showing up with the Servlet transport.

Then came Jitr, the latest creation of international software samurai Josh Devins:

Jitr (pronounced "jitter") is a JUnit Integration Test Runner. It allows your web application integration tests to easily run against a lightweight web container in the same JVM as your tests.

It happens Jitr is extremely well fitted for functionally testing a Mule instance embedded in a web application. Here is the structure of such a Jitr powered test:

<br />@RunWith(Jitr.class)<br />public class ServicesFunctionalTest {<br /> @BaseUri<br /> private String baseUri;<br /><br /> private HttpClient httpClient;<br /><br /> private MuleContext muleContext;<br /><br /> @Before<br /> public void initializeTest() throws MuleException {<br /> httpClient = new HttpClient();<br /> muleContext = MuleServer.getMuleContext();<br /> }<br /><br /> @Test<br /> public void pingService() throws Exception {<br /> final HttpMethod method = new GetMethod(baseUri + "ping");<br /> httpClient.executeMethod(method);<br /> assertEquals("PING: OK", method.getResponseBodyAsString());<br /> }<br /><br /> ...<br />}

Thanks to the access to the MuleContext, all the internals of the loaded Mule instance are available, which means that I am free to use my asynchronous testing strategies. It's party time. really!

Note that Jitr tests are pure JUnit 4 tests, without any sub-classing needed.

Because I am still loading a slightly different configuration at test time, as I am replacing file endpoints with VM ones for easier testing, I have to load a different web.xml file. Nothing to worry, as Jitr offers the possibility to load this file from an alternate location by means of configuration only. So here is the actual beginning of my test class:
<br />@RunWith(Jitr.class)<br />@JitrConfiguration(warPath = "src/it/resources/webapp-it")<br />public class ServicesFunctionalTest {<br/>...

Jitr is a capable new tool that will complement the tool box of any developer having to run functional tests on web services.

Legacy Tests

In Working Effectively with Legacy Code, Michael Feathers states that, for him, "legacy code is simply code without tests". As he says it himself, he has "gotten some grief for this definition". I can understand why.

I have come to deal with some legacy code monsters recently and came to realize that this code had tests written for it. So could it be that this code was not legacy after all?

After a closer inspection, the truth finally came out:

• the test coverage is very low (less than 30%),
• the tests explore counter-intuitive paths (for example, just the unhappy ones),
• the tests themselves are monstrous (created in what looks like a copy/paste spree),
• they overuse mocks in a lax manner (many are not verified at tear down),
• they do not verify the complete state after execution (no validation of indirect outputs like request and session attributes),
• and, corollary of the legacy framework the code targets (EJB 2), full correctness can only be proven when deployed in an application container.
  

I immediately thought of chapter 9 of Uncle Bob's Clean Code, where he advocates that unit tests should be written with the same care than for actual code. The tearful hours I spend refactoring these tests so I can then refactor the code they are supposed to exercise, are a good testimony to this imperious necessity. Keep tests clean!

Else you will end up with nearly worthless legacy tests.

JBound 1.1 is out

I have just released version 1.1 of JBound, my simple utility that performs boundary checks on domain model objects.

This new release features native support for more JDK data types and the possibility to register custom data types.

Consult the daunting one page user guide for more information.

ESB Testing Strategies with Mule

SOA World Magazine has just put online my latest article, "ESB Testing Strategies with Mule", which they have published in their August issue.

Commit Risk Analysis

I like to compare the stability of a legacy application as a saddle point:

QA provides some lateral stability that prevents the legacy application (this little funny red dot) to fall sideways. But, still, a little push on the wrong side and down the hole the application will fall.

What the legacy code typically misses is unit tests. With unit tests, the saddle disappear and the code stability ends up in a pretty stable parabola:
In an ideal world, there will be no legacy code. In a little less ideal world, there will be legacy code but every time a developer would touch any class, he would first write unit tests to fully cover the code he is about to modify.

Unfortunately, in our world, code is written by human beings. Human beings turned the original code into legacy code and human beings will maintain and evolve this legacy code. Hence there will be modifications on non unit tested code that will end up checked in. Inevitably, the little red ball will drift on the wrong side of the saddle.

I came to wonder what could be a way to estimate the risk that has been introduced by changes in a code base. Basically, the risk would be inversely proportional to the test coverage of the modified class.

I came with a very basic tool, Corian (Commit Risk Analyzer), that simply fetches the Cobertura coverage percentage for classes that were modified in Subversion for a particular number of days in the past.

Do you know of any method or tool for estimating the risk that a series of revisions could have introduced in a code base?

Its name is JBound

I needed a little tool to torture with boundary values the public domain model objects of a project I am working on. There are excellent test generator software like Jtest or late AgitarOne, but I needed something really scaled down that would act as a complement to existing unit tests.

There came JBound, a mere six classes utility, whose sole purpose in life is to inject unfriendly values, like extremes and nulls, in the constructors and setters of a class. It also calls getters and standard methods like equals, hashcode and toString, just to be sure that the created object is not completely ravaged inside.

Using JBound is easy:

    @Test
public void fullyExerciseBeans() {
    JBound.run(new Exercises() {
        {
            forClasses(MutableBean.class, ImmutableBean.class);
        }
    });
}

By no means, JBound should be used as an artificial mean of creating high test coverage. You still need to write sensible tests, especially for methods like equals, if your object identity is strictly controlled. Moreover, it is debatable if JBound is useful for non-public APIs, as internal classes are less prone to be exposed to end-user abuses.

Is JBound original? Not at all! But, as I have been baffled by the amount of glitches this tiny tool has found in my project, I thought it could be interesting to share it. Maybe JBound will prove useful to you to. Maybe you will even feel you can make it better. If this is the case, please explain me how and I will welcome you aboard!

A Zephyr is blowing

My attention has been recently drawn to a new tool named Zephyr, which dubs itself as "The Next Generation Test Management Tool".

My very first impression is that the Zephyr team has done a great job putting on-line a complete workable demonstration environment. This is great to quickly delve into what the software is really about and getting assured that "it works on my machine", which is crucial nowadays as the corporate IT landscape is much more diverse than the traditional Windows + Internet Explorer desktops environment it used to be.

As far as platforms are concerned, Zephyr seems to run "on standard Windows desktop" (sic). I reckon they meant "Windows server", though I have seen production systems on desktops! Hence, I could not test the ease of install nor investigate the technologies used. Neither could I estimate its capacities to scale or work across a WAN.

The second impression is that the graphical interface is compelling, if not mind blowing. These guys did a great job of making arid forms filling an almost bearable task. Indeed the tool is brainy enough to avoid manual data copying and is able to pre-fill or filter data according to the context of the task.

Zephyr is also smartly aware of agile principles, as the notion of Scrum's sprint is hard wired into its dashboards. And dashboards is where Zephyr really shines. I love dashboards of all sorts, even complicated ones. But Zephyr's are truly awesome:

The dashboards and workspaces are tailored to the user profile, which makes navigation easier because you do not have to filter out a lot of non-relevant features. The tool does a great job at integrating and aggregating all sorts of QA related data, including the ones coming from defect tracking systems.

And I think this is where one of the challenges Zephyr will face resides. It currently connects to Bugzilla only, but there are many others out there. Moreover, companies have developed an habit of using their defect tracking system as a management tool for QA. How is Zephyr going to convert these users to this new platform? How disruptive for the practice would it be to move from the use, say, of JIRA to using Zephyr?

The thing that really truly bugs me about this tool is that it does not go further than the traditional "QA monkey" work, in which a human beings are presented a list of actions to perform and report the results thereof. That a tool with an agile penchant does not incite people to evolve their QA practices to more automation is flabbergasting. Where is the Selenium integration? Where is the FIT connector? Though manual QA will never be fully replaced, at least supporting a blend of automated and non-automated tasks would be a great first step. What the software industry really needs is lazy QA teams who mainly use their brains to work on automating their tasks!

To finish on a positive note, Zephyr takes integration seriously. It exposes JSON and REST APIs, the assurance that if you opt for this tool, you will not end up with yet another instant legacy application. This is something I would like to see more in so-called enterprise grade applications!

If you are looking for ways to improve your QA management, I can only recommend that you get Zephyr now, as the free 3 users licenses will allow you to see what this "Next Generation Test Management Tool" can do for you.

UPDATE 31-MAY-2009: Zephyr version 2.5 now comes with test automation features that include the ZBot technology.


ZBot allows you to execute testing scripts on remote machines and aggregate all results back in Zephyr. This is a great move, which addresses my concerns about the lack of support for automation in the previous releases.

Unit Tests: No Future?

Industry expert Andrew Binstock has just posted an entry in his blog titled "Is the popularity of unit tests waning?", where he discusses the staggering state of the practice of unit testing.

Andrew asks the crucial question of why did we end in this situation. I do not pretend to have the answer, but here are some patterns that I have observed, which, I think, could decrease the appeal of unit testing.

Bad unit testing practices

It is very easy to write fragile unit tests, for example by using stubs when mocks would be enough or by creating time sensitive tests that work intermittently or stop working after a while. Similarly, it is common to see integration-like tests sleep into the realm of unit testing, making these tests fragile, slow and dependent of external resources.

These bad practices tend to lead to a decrease of confidence in unit tests, which then can lead to a progressive reduction of their usage. For example, one programmer can decide he will not write any tests anymore for DAOs after struggling with poorly written tests.


Bad software design

There is a tight relationship between good code design and testability. I have already blogged about how increasing test coverage can lead to a better design: unfortunately, not all developers are ready to review their design to make their code more testable.

To be fair, our languages and frameworks often force us to write code that is hard or uninteresting to test. Who wants to write unit tests for infamous JavaBeans getters and setters? Not Allen Holub, for sure!

Other programmatic idioms like equals/hashcode often exhibit a high cyclomatic complexity: writing complete tests for those would be tedious, unless one use test generating products like Agitator, from late Agitar, or Jtest, from Parasoft.


Test benefits blindness

Management tend to be blind to the benefits of unit testing and very aware of its costs. A project I know have been deemed to have gone overboard with unit testing. At the same time, this project has the code base that is the most maintainable, flexible and fun to work with. It seems something prevents businesses to see the value added of solid test practices, as if it was all about some obscure geeky self satisfaction activities (as writing "perfect code" would be).


No Future?

Is there any hope, then? I think that, like for most of our problems on this little planet, education is the key. Whether we look at software engineers or business managers, there is still a great lack of education about unit testing, how it works and why it pays back.

Bugs Of Opportunity

Yesterday night, I fixed a trivial bug in NxBRE. Doing so, I have spent almost 5 times more time writing tests to assert the current behavior and the expected one when the bug would be killed.

This reminds me of an earlier reflection on how being test infected changed my reaction to incoming bugs. Prior to become green light addicted, bugs were my enemy and were received a treatment depending on my mood:

They are now an opportunity to improve and increase test coverage, regardless of the mood I happen to be in:
But, as you have certainly noticed it, grumbling is still part of the process!

Agitation On Demand

In my previous post, I talked about how sweet it is to push the limits of unit tests for the sake of increasing test coverage. During the testing frenzy I mentioned, I came to test a class whose most of the methods throw an UnsupportedOperationException.

Ugly? Well, this class is a subclass of javax.servlet.jsp.PageContext with a very narrow scope of usage, so this is why.

I wanted to generate calls to all these methods and catch the exception as the expected test result. My first thought was to use reflection to do this in a dynamic manner but before to opt for this approach I decided to Google for free unit test generators. This is how I came to discover JUnit Factory, a test generation service from Agitar. I already knew Agitar from Agitator, their award winning unit test generator. But their online generation service was unknown to me.

Needless to say that this service is very well done. The client I used is an Eclipse plug-in, that seamlessly installs on version 3.2 from a proper update site. To avoid uploading my whole project to their service, I isolated my troubled class in a temporary project and pushed the magic button.

A few seconds later, a test case was automatically added to my project, with all my problematic methods (and all the other ones) thoroughly test covered. Awesome: agitation on demand truly works! In other times, anarchists would have rejoiced.

When I decided to include this test case in my main project, I started to hit some issues. The unit test does not extend JUnit's TestCase but an Agitar custom one. Decided not to be stopped by such a minor issue, I promptly added their library to our Maven repository.

Then I pushed the execute button in Eclipse to run my tests and the Agitar test case complained because the launcher was the standard one from JUnit and not their own. This is when I called it quit. Of course, I could have modified my Eclipse start command to use their launcher. But with my Maven builds and reports?

That was a little too much to do and I decided to walk the path of reflection to systematically invoke the targets methods.

This is too bad because JUnit Factory is really a great concept. Try it, it might work very well for your needs.

Push Tests, Pull Cover !

I have nearly spent the whole day increasing the test coverage of one of my project. For traditional employers or ill educated project managers, this could sound like an incredible waste of time. For others, this could sound like pure geeky zealotry or a new mean of self satisfaction for IT nerds.

Indeed, seeing the coverage report display numbers above 80% is extremely satisfying but the true reason for this testing frenzy is elsewhere: this project being bound to play a critical role in the core of the business of my company, testing it thoroughly is simply the only option.

Beyond the assurance such a high coverage buys, the implicit documentation of the system's behavior that an extensive battery of test creates is worth the effort on its own. But there is more. As recently discussed by industry guru Andrew Binstock, writing tests will inevitably lead you to question both design and implementation. Hard to test classes full of not covered "blind spots" will cry for refactoring!

I was amazed to see my code getting simpler and more concise while I was struggling to push the limits of the test coverage. I wish you similar amazing moments!

Seriously Infected

After writing some tests for a Tomcat valve today, I came to wonder about what is it that I like the most with unit testing.

At first, I thought that it is when these red lights turn green - oh the jolly green! This is a truly enjoyable moment when all tests pass.

Then I thought, it is maybe when green lights turn red - oh the scary red! When I touch some piece of code and immediately see the impact it has, I really feel like unit testing saved my day (and possibly some nights).

But finally, I came to realize that, for me, the best moment with unit testing is when I ask myself this simple question: "How am I going to test this?". No matter if I am testing an existing piece of code or writing the tests first, asking this question is really a delightful moment. I reckon this is because I start considering what I will test as a living entity and not anymore as a mere concept: by figuring out its execution environment, its inputs and outputs, all this code or code-to-be comes to life in a very vivid way.

Am I describing the apex of some kind of geeky childbirth? I do not know but what I know for sure is that the more complex it is to imagine how to test something at first glance, the most rewarding it will be to figure out how to do it!

Put Your Business Rules to the Test

In the November 2006 issue of DDJ, Scott Ambler explained in an article titled "Ensuring Database Quality" the why, what and how of database testing. With the advent of commercial and open source business rules engines and their increased usage in applications of all size and complexity, ensuring rules quality has become a critical issue for many businesses.


No blind spot tolerated

There are several reasons why applications externalize their business rules instead of hard coding them, including the following:

• to accommodate the need of frequently changing business rules in the cleanest possible way,
• to enable non technical personal to author business rules,
• to allow the exchange of rules between heterogeneous systems,
• to delegate the handling of complex rules to a component trusted for its performance and reliability.

A successfully implemented rules engine will quickly end up handling the most critical aspects of a modern business infrastructure, as its strategy will become reified in rules and enforced by the engine. Therefore, the need for properly managing these rules will quickly become compelling.

Whether they use commercial tools (that come complete with advanced tooling) or open source implementations (that are traditionally weak in term of tooling), implementers will have to put in place a rules life cycle similar to the one shown hereafter.


One of the critical aspects of rule management we are going to discuss here is testing. Since we recognize what is at stake with business rules, keeping them in a blind spot and simply hoping for the best will have sooner or later direct and dramatic consequences that are likely to hurt the bottom line for the aforementioned reasons.

We will discuss the different steps necessary to properly set up a test environment and then how to iterate on this basis.

The specific case of the home grown engine

If the rule engine you use is home grown, it is essential to extensively test it to certify that, release after release; it keeps its deduction power! To ensure this, a combination of component level unit testing and white box testing is needed. They will both rely on a specific set of data and rules, usually not actual ones but ones made to cover core specifications, well known technical challenges (like critical values handling) and regression testing (tests created after bug reports).

These tests should also be completed with performance and load tests, in order to guarantee that your engine remains in its defined production constraints.

Setup Step 1: Setup the test sandbox

Rules engines are generally external components designed to be fed with data they can process and make deductions on, which usually end up by updating the existing data or by creating new data. This relative isolation, which somehow plays against the engine at runtime because it is costly to alleviate, can be leveraged for testing as it simplifies the setup of a sandbox, i.e. a dedicated and segregated environment where the engine will be free to modify data without touching any actual critical information.

For engines that are directly connected to enterprise resources like databases, the setup recommendations in Scott's article will apply.


Setup Step 2: Build input reference

Unless your business rules are simple or the amount of information you are dealing with is limited, it is almost impossible to manually create sample data that will represent the diversity of cases your engine will have to process and apply rules on. If you are in this situation, you will then have to work with business analysts to select relevant batches of data that represent a reasonable input in term of data complexity and diversity. This will become your input reference at one point of time.


Setup Step 3: Build output reference

After building this input reference, you will have to make your engine apply rules on them: the result will become your output reference, but only after having being validated by an expert.

With this reference data in hand, you will be in the position to certify your rule base, which is usually done by versioning it. It is of paramount importance that not only the rule file(s) be versioned but also all the reference data and related artifacts (like configuration files). This bunch of files will form a consistent, tested and validated set of information that will be candidate for production and, if need be, restoration if a downgrade is needed. It will also be the unit of data that will be copied when a new version of the rule base will be needed.


Following Steps: Iterate

After this initial phase, each subsequent modification of the rule base will have to go through a similar validation process, with the difference that the input reference will probably need edition to include new business cases or remove irrelevant ones.

When testing the modified rule bases, differences with the output reference will be induced by the simple fact business rules have been modified (for example, if a discount rate has been lowered, the new computed values will change). Again, a business analyst will have to analyze the difference between the previously validated output reference and the new candidate. A diff-like visual tool with a simple accept/reject feature could be of great help here.


Merry nights ahead

Business rule engines are now first class citizens in the agilist's toolbox. As such, they deserve a well defined and thorough test strategy. Depending on the engine used, setting up such a strategy will vary between defining your own or embracing the one provided by a vendor. At the end on the day, what really matters is what testing buys you: the possibility to make the most of business rules engines, that are intrinsically able to embrace changes, without losing your sleep.

Testing Toolbox Gets Richer

ZePAG has released the first version Usurper, a neat tool for easily creating test instances of your value objects. It is a nice compliment to the classical mocking and stubbing tools.

I warmly invite you to read his post presenting Usurper and then to rush to the tool web site to start benefiting from its goodnesses right away.

And please, do not forget to ask him why he opted for the org.org package root!

Coverage Generation or Generations of Coverage?

While I was thinking of this post, I came along the "32 & 16 Years Ago" section of the July 2006 issue of Computer and was amazed by what I read in the entry titled "Software Testing". It started like this: "The objective of (...) probe insertion mechanisms (...) is not to find errors; rather it is to quantitatively assess how thoroughly a program has been exercised by a set of test cases" (edited by Computer).

This is dated... July 1976! Code coverage in 1976! Golly! What a blunt reminder that we are, at most, building on the shoulders of giants... Humility is definitively the only path that is safe to walk.

Anyway, my current concern with code coverage was to fix the 100% target that I was talking about in my "Programming for Agility" post and that I borrowed from Uncle Bob himself. This is a nice high level target but can it really be reached? I do not have much doubts for code I am writing with my little busy hands but what with generated code? Especially code that contains many conditional branches like the one Eclipse generates for equals() and hashCode() methods?

@Override

public boolean equals(Object obj) {
if (this == obj)
 return true;

if (obj == null)
 return false;

if (getClass() != obj.getClass())
 return false;

final DefaultTodoListItem other = (DefaultTodoListItem) obj;

if (id != other.id)
 return false;

return true;
}

I came with three possible strategies, including one for which I already apologize:

1. Exclude generated code from test coverage estimation: for having written this one, I have already slapped my hand as a due punishment* ! Generated code does not mean intrinsically safe code: it is live code and is subjected to all sorts of possible errors you can imagine.
   
   
2. Use test code generators: great tools like Agitar Agitator can very conveniently generate test cases for all sorts of extreme values that will most probably explore all the possible execution paths of the aforementioned generated methods. These tests would complement nicely the ones written by the developers.
   
   
3. Reduce the number of needed tests by enforcing class invariants: this can be done by working with immutable objects and check for invalid or exceptional conditions at construction time. This also allows to pre-calculate values like hashCode() and toString() rendering. The tests will then target user written code and will exercise construction of invalid objects instead of testing the calls of all methods possibly sensitive to an invalid state object.

Please do share other possible approaches or practices you think of or daily use in this matter. Even if we are the second generation of code coverage buffs, we might still have something useful to say!

____________________
* An efficient self-punishment technique I learnt from J.B. Rainsberger during SD West 2005.