In response to “The Oracle Problem and the Teaching of Software Testing”

In The Oracle Problem and the Teaching of Software Testing post, Cem Kaner wrestles with teaching how a tester could determine oracles for an exam question about automated testing. Unfortunately, the window for posting replies on the post has expired. Thus… this blog post is born! :)

Here’s the exam question he gives his students:

Suppose you have written a test tool that allows you to feed commands and data to Microsoft Excel and to Open Office Calc and to see the results. The test tool is complete, and it works correctly. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use to help you find bugs? What types of information would you expect to get with each oracle?

Note: Don’t just echo back a consistency heuristic. Be specific in your description of a relevant oracle and of the types of information or bugs that you expect.

As other comments on the post point out, I also find Hoffman’s list better suited for this exam question. As Cem Kaner already mentioned, they do indeed lend themselves well to being coded. After all, that’s exactly what they were designed for.

With respect to teaching a tester to generate oracles for the question, perhaps the student might be confused by the question itself. I don’t find it straight-forward to think about what oracles I would be using before I think about what information I’m trying to find out about the new version of Calc. As a tester, how often do we think about the oracle before thinking about what information we’re trying to ascertain?

As an example, perhaps I’m interested in finding out more information about Calc’s implementation of the SUM function. I may have used the Hoffman list which triggered a thought about the reference program oracle. Maybe this lead me to using the command feeding test tool to compare with Excel. Or perhaps Bach/Bolton’s comparable product heuristic triggered a similar thought. Why not use the “consistent with itself” heuristic to use Calc as its own oracle? After all, SUM should have an equivalent result when using a Calc addition operation? Would I have missed this if I were using the Hoffman list? Would I have been preoccupied with using Excel as a reference program rather than Calc itself?

In my own test design experience, I don’t find that the Bach/Bolton list triggers as many ideas as let’s say the SFDIPOT list or even just a giant list of test techniques/objectives (function testing, scenario testing, compatibility testing, etc.). Although I’m not as familiar with Hoffman’s list, you can bet I’ll be making use of it now. As Cem and other commenters on the post have mentioned, each list is not complete. Once we know what information we’re looking for, the Bach/Bolton list, Hoffman list or some custom combination of lists could be used to trigger thoughts about useful oracles which may or may not be automated.

2 thoughts on “In response to “The Oracle Problem and the Teaching of Software Testing”

  1. Hi, Ray:

    I’ve reopened comments, so you are welcome to post something on the blog…

    I agree that this is a difficult question that requires the student to think about their information objectives. In this course, I hand out a list of potential exam questions on the first day. Students have a month to prepare for the exam, and during that time we spend a lot of time on information objectives and other topics that are fundamental to the design of a testing strategy. Questions like this ask students to integrate ideas across a few lectures instead of parroting back an idea from one narrow part of the course.

    I see effects of this type of difficulty (a demand for integrative study and response) throughout the students’ exam answers. Making students cope with this difficulty is one of my instructional objectives. This level of thinking by the student yields better understanding, better retention, and greater likelihood that they will actually use what they learn later.

    What I finally, and reluctantly, concluded is that the students’ specific difficulties with this question showed an additional failure pattern–they were being seduced by a superficially attractive way of describing oracles that could give them ideas for effective bug reporting, but wouldn’t give most people good ideas for test design, especially automated-test design. That realization is what motivated my post, and the pending design change for the course.

    – cem

Leave a Reply

Your email address will not be published. Required fields are marked *

seventeen + eleven =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>