It’s almost Christmas. I live and work in Sydney, and this time of the year is pure madness around here. I’m guessing it is madness everywhere in the developed world, where the meaning of Christmas seems to be purchasing as much stuff as you can, as close to Christmas day as possible. People seem so desperate, and is it just me or it feels like the world is coming to an end? I wish I could blame the Mayans, but every Christmas is the same – not only this one. The spirit and meaning of Christmas is totally lost in the madness.
This insane time of the year reminds me of metric madness or defect madness. Every tester has been through it at least once, but more likely several times. It is when testing becomes a numbers game, when it loses its core purpose and value. When it is all about the number of defects you raise, the number of test cases executed the amount of reports sent. Metric madness happens when testing is no longer about reporting quality or about finding important information abou the software, when it is all about the numbers.
Seeing developers and testers spend hours classifying defects, discussing if the defect is a SIT or UAT defect, or if it is really a severity 2 and not a severity 3 almost brings me to tears. If all that effort and time was spent on exploring, learning about the system and providing feedback wouldn’t we deliver better software or at least better information to decision makers?
Or all that wasted time in trying to ensure we execute the ‘correct’ amount of test cases each day, to show that we are on track. Testers that take their craft seriously know that numbers mean nothing, that numbers can be manipulated and misunderstood and worse, they can mean different things to different people.
I am in the midst of a project going mad with metrics. I’m trying to change it, one report at a time – however it is not an easy process. To change people’s minds, to explain the obvious, that test case execution percentage doesn’t mean much is a long arduous process. However as a context-driven tester, that is the only path that is worthwhile, productive and, above all, ethical.
And on top of that, metric madness reaches a boiling point when both testers and developers’ output is measured by these same metrics (such as number of defects raised). It is not only insanity, but it is unproductive and it destroys teams, creating huge chasms between teams.
As I write this I wonder, how can something as plain as the nose on one’s face, a truth that is so obvious to some, be so hard for others to understand. How can that be? One thing I know, us ‘sapient’ testers out there, have an obligation to work towards stoping the madness. The real question is: how can we do that faster and more effectively. Well, I reckon that will have to be the topic for another post!
If you would like to read more about testing metrics, these two blogs are a great place to start: Why pass vs. fail rates are unethical and Metrics, Ethics & Context-Driven Testing.
Road Less Tested 