When More is Less

I often find myself having the same frustration conversation about test resources, often with different people, but sadly sometimes with the same person.

If you are a seasoned tester, I’m sure you have come across this situation before. You and your team started test execution and a few weeks into execution mass panic hits project members and everyone tries to offer a solution on how to improve the ‘efficiency’ of test execution.

Although most people have the best of intentions, the constant ‘are we there yet’ question is not really helpful.

Far too often development gets delayed and testing gets squeezed as the delivery date remains unchanged. I find the closer we are to the finishing line, the more people try to ask if they can do anything to help getting testing over the line. That is when the question ‘can we add more testers’ to help execution starts getting asked.

I find this like of questioning troublesome and there are a number of fallacies that need to be addressed when answering such questions. I’ll address two in this post:


A person who asks this type of question often does not have the slightest understanding of what testing is really about. They have this idea that ‘anyone can test’ and that testing is just a matter of following a set of steps – hence the more people we add to the execution phase the better right? Wrong! Oh so wrong.

I’m not going to get into the argument that good testing is so much more than following a set of steps, as I believe this topic has been covered in depth in the blogs of several prolific testers and industry leaders. All thinking testers agree that this is not the case, that testing is in fact an intellectual process, instead of a mechanical, repetitive one. However the ‘anyone can test’ theory is not the only issue with the ‘lets add more testers’ proposal.


There is a fundamental principal in economics called the law of diminishing returns. In a nutshell, it states that at first, by adding more people to any given task will increase productivity, however (all else being equal) it will get to a point when the more people added to the task the less productive the task will be.

“Consider a factory that employs laborers to produce its product. If all other factors of production remain constant, at some point each additional laborer will provide less output than the previous laborer. At this point, each additional employee provides less and less return. If new employees are constantly added, the plant will eventually become so crowded that additional workers actually decrease the efficiency of the other workers, decreasing the production of the factory.”

The concept is quite simple and in testing it manifests itself in many different ways:

Experienced testers become trainers – and are taken away from hands-on testing as new testers are added to the team. The productivity of these highly important team members is impacted, affecting the productivity of the entire team.

Duplication of effort – for example, when too many testers are testing the same functionality, the same defect could be raised more than once. Or if an area of the system is blocked due to a defect, many testers are impacted.

Surrogate testers – another problem is that often these ‘testers’ are not testers at all. They are other project members (business analysts, developers) pulled out of their own day-jobs to execute a set of test cases. Some of these team members have no interest in becoming a tester, and therefore do not have the mindset necessary to conduct good testing, which brings me back to the anyone can test fallacy. It is sort of a cycle.

Law of Diminishing Returns

There are however, some instances where more testers will improve throughput and productivity of the test team. As with anything in testing the right answer will depend on the context of the project and the team. After analysing my team’s structure and the project’s status if adding more testers will be more of a hinderance than a help, I often use the arguments above to initiate discussions.

Metric Madness

It’s almost Christmas. I live and work in Sydney, and this time of the year is pure madness around here. I’m guessing it is madness everywhere in the developed world, where the meaning of Christmas seems to be purchasing as much stuff as you can, as close to Christmas day as possible. People seem so desperate, and is it just me or it feels like the world is coming to an end? I wish I could blame the Mayans, but every Christmas is the same – not only this one. The spirit and meaning of Christmas is totally lost in the madness.

This insane time of the year reminds me of metric madness or defect madness. Every tester has been through it at least once, but more likely several times. It is when testing becomes a numbers game, when it loses its core purpose and value. When it is all about the number of defects you raise, the number of test cases executed the amount of reports sent. Metric madness happens when testing is no longer about reporting quality or about finding important information abou the software, when it is all about the numbers.

Seeing developers and testers spend hours classifying defects, discussing if the defect is a SIT or UAT defect, or if it is really a severity 2 and not a severity 3 almost brings me to tears. If all that effort and time was spent on exploring, learning about the system and providing feedback wouldn’t we deliver better software or at least better information to decision makers?

Or all that wasted time in trying to ensure we execute the ‘correct’ amount of test cases each day, to show that we are on track. Testers that take their craft seriously know that numbers mean nothing, that numbers can be manipulated and misunderstood and worse, they can mean different things to different people.

I am in the midst of a project going mad with metrics. I’m trying to change it, one report at a time – however it is not an easy process. To change people’s minds, to explain the obvious, that test case execution percentage doesn’t mean much is a long arduous process. However as a context-driven tester, that is the only path that is worthwhile,  productive and, above all, ethical.

“Metrics that are not valid are dangerous”

And on top of that, metric madness reaches a boiling point when both testers and developers’ output is measured by these same metrics (such as number of defects raised). It is not only insanity, but it is unproductive and it destroys teams, creating huge chasms between teams.

As I write this I wonder, how can something as plain as the nose on one’s face, a truth that is so obvious to some, be so hard for others to understand. How can that be? One thing I know, us ‘sapient’ testers out there, have an obligation to work towards stoping the madness. The real question is: how can we do that faster and more effectively. Well, I reckon that will have to be the topic for another post!


If you would like to read more about testing metrics, these two blogs are a great place to start: Why pass vs. fail rates are unethical and Metrics, Ethics & Context-Driven Testing.

Stand up for good testing

An incident in a Melbourne bus yesterday really got me thinking. A woman was verbally abused over and over again inside a crowded bus and almost no one attempted to intervene. A bus full of upstanding citizens just sat (or stood) there, right next to the victim and everyone just let it happen. What is shocking to me is that two more people joined in the verbal tirade.

This is a classic example of mob mentality. However why didn’t the mob mentality work on the majority of other passengers there, who were appalled by the attack? Why didn’t anyone say anything to defend the woman? The obvious reason is fear, and in this situation it was probably not safe to say anything as all three attackers seemed aggressive.

However there are other situations when lives are not in risk when people still do not standup for what is right. I find that it happens in testing happens far too often. There are good testers in companies everywhere, who know there are better ways of testing things, more productive or cost effective ways, however they come to work everyday and do nothing. Some do nothing because they are OK to work their day jobs and go home, they are just not engaged or interested. However there are those who would like to change, that are interested enough in their profession to research, read and discuss testing, but they also do nothing because in their test team no one else wants to hear about it.

So, just like the people in the bus, who felt horrible for the woman being abused, these testers do nothing because confrontation is uncomfortable, especially if you are unsure if your peers will back you up, or will hand you out to dry.

I will paraphrase a quote by Edmund Burke and say that ‘All that is necessary for the perpetuation of bad testing is for good testers to do nothing’. It takes a special type of person to speak up for what they know it is right. In any situation.

Zombie Slayers

Rise and Fall of Zombie TestersYesterday I presented at the Girl Geek Meetup in Sydney. For the first time Girl Geek Meetup presentations followed ‘Ignite’ format, inspired by Ignite Sydney, where each presentation goes for only 5 minutes. Each presentation contains exactly 20 slides, with the catch of auto-forwarding slides every 15 seconds. It sounded easy enough to come up with a topic and speak for 5 minutes, however the auto-forwarding slides proved to ve a bigger challenge than what I first anticipated!

To me the hardest part was having to keep up with the slides. Often when presenting one draws on the audience response, and tends to spend more time on topics that seems to be interesting to them. With this format of presentation, the presenter cannot afford such luxury. Fifteen seconds is all you have.

What attracted me to this presentation style is that presenters have to go straight to the point, there is no time to beat-around-the-bush or to embellish specific points. And presenters need to be prepared. There is nothing worse than turning up for a presentation just to find the presenter reading through their notes, unprepared. Finally, as slides only show for 15 seconds, slides have at most one phase or paragraph. Most have only pictures and titles. That makes presentations much more interesting and avoids the dreaded “death by powerpoint” syndrome!

Something else that made me chose to present was the opportunity to speak about testing to an audience of mostly developers. In my opinion, testing gets a bad wrap in the industry and I jumped at the opportunity to try to speak to developers that not all testers are zombie testers.

My presentation’s title “Rise and Fall of Zombie Testers” seemed to catch people’s interests, so I was off to a good start. I was a bit worried with the line up, as I was the seventh presenter. I thought by the time I took the stage people would be half asleep. However, thanks to the Ignite style of presentation, everyone was still very awake and interested when I presented.

My main objectives were to explain to the Girl Geeks that not all testers are the same – that some of us are thinking testers. That we take pride in what we do and we approach testing and QA as a craft form, which we are constantly trying to improve and perfect. That we are zombie slayers – in the sense that we try to educate testers everywhere about good testing, hence eliminating zombies, one at a time!

There was a very positive response to the presentation! Some of the developers were very interested about the concept of thinking testers, and many of them had never heard of it.

So, zombie slayers everywhere, unite and get out there!! I am doing my part, educating developers and testers – what are you doing to spread the word?



Every so often I read a book that really changes the way I think or see things. A while back I read The Medici Effect (Frans Johansson), a book about stepping into the Intersections in life. Johansson describes intersections as places where ideas from different cultures and fields meet and collide, ultimately igniting an explosion of extraordinary new discoveries. It is a book about exploration and the art of maintaining an open mind. It entices the reader to keep curious in a world with infinite possibilities.

It is uncanny how well that translates to testing software. Testers should keep curious, should explore obscure paths that may lead to nothing, but that ‘gut-feel’ says otherwise.

One of the intersections that interests me the most is the intersection between psychology and software testing. In the classic testing book Lessons Learned in Software Testing (James Bach, Bret Pettichord, Cem Kaner), there are many mentions about the psychology of testers. For example, testers (and anyone else really) are biased, and are more likely to pay attention to test results that confirm a tester’s own opinion of the product, developers, organisation, etc – this is called Confirmation Bias. There are numerous other bias mentioned in the book. It goes to show that if you know a little about psychology, you are more likely to be a better tester.

It pays to stay curious, to explore things that may seem unrelated. In the worst case it will make life more interesting, in the best case you may step into an intersection moment that might just change everything!

Stay tuned, I have a feeling I’ll be blogging a few more times about intersections in testing!