Initial look at automating your testing

Why QA needs to change?On Martin Anderson’s blog, he wrote a post Why QA needs to change? A few passages struck a chord with me and I considered posting a reply within the post itself. Due to the length of my response, I’ve chosen to publish a blog post instead.

In Martin Anderson’s Why QA needs to change? post, he wrote:

…The ability to deliver code into production at will has a direct effect on your bottom line but to do this effectively you need two things, 1) understand the important areas of functionality that the customers really use and 2) be able to test these areas as quickly and easily as possible.

The first is a business issue but the second boils down to automating your testing.

I agree with all points, except I do not agree that it always boils down to automating your testing. I agree understanding the important areas of functionality and making testing faster and easier is the right direction. I agree automating certain aspects of the testing process can help speed up parts of the testing process. I agree automation certainly seems to make it easier to run the automated aspects of the testing process.

However, we’d probably benefit from redefining what “automating your testing” means. To me, it means automating the verification of facts that we know about the system under test (SUT). The test code arranges the system, an action is performed and then one or more assertions (facts!) is made about the state of the system. Or… given a certain state of the system, when some action is performed, then certain facts about the system are true.

However we spin it, running this automation is not testing, it’s just verifying facts that we believe should be true when the system is behaving as expected. What’s testing?

  1. Designing the test code to perform the verification. That’s testing.
  2. Evaluating the results of the verification to determine whether a failing verification is a bug in the system or a bug in the test code. Yes, that’s testing.
  3. Redesigning the test code so that a failing verification provides more useful information to the individual evaluating it. That’s testing.
  4. Making the determination that a failing verification is actually a bug in a programmer’s interpretation of an ambiguous requirement. That’s testing, too.

Whether the act of designing, evaluating, redesigning and requirements follow-up is done by a programmer, tester or anyone outside these roles, it’s testing. At this moment, these testing activities are very difficult if not impossible to automate.

Going back to the original goal of delivering production code at will, rather than prioritizing by areas of functionality that the customers really use and then focusing your testing on those areas, why not focus on those areas that represent the biggest risk to the business value of the software for the customer?

If you’re unlucky, these might be the very same areas. More likely than not, the riskiest areas are a subset of the functionality that the customer uses plus any other new areas that might present new risks. These are the areas that testing should focus on.

Re-running that tired, old test on an area of the system that hasn’t changed but was deemed an “important area” tells us virtually nothing new about the system and adds no value to the testing process.

However, targeted testing (using manual or automated verification) on those high risk areas (areas that have changed recently, areas that have been the source of numerous bugs, areas that were recently added, etc.) is one shortcut to delivering high quality testing as quickly as possible.

Will this allow testing to happen more quickly? Probably. This approach allows testing to zero-in on the riskiest areas to find important issues quickly. It won’t find all issues (that’s impossible!), but it should give feedback to the team faster than attempting a singular focus on automating as many of the important areas as possible.

Will this be easy? Probably not, but testing is not easy either. Yet another reason why testing is so difficult to automate.

Leave a Reply

Your email address will not be published. Required fields are marked *

9 − 9 =