Cem and James talked about exploratory testing — Cem talked about exploratory testing in terms of categorizing knowledge — by risk, test technique, or approach. Though I wasn’t there, from other comments (and talks with James), I would guess he talked about the use of heuristics and skeptical questioning of the application. He gave the URL for the Satisfice Heuristic Test Strategy Model. This model presents 32 perspectives in 4 areas (though Cem makes it 5 areas, calling risk out separately).

Back to Cem — people (such as Giri Vijayaraghavan) have made taxonomies of various bugs that could exist in a certain type of application (such as shopping cart apps, available at http://www.testingeducation.org). He also talked about the use of “attacks” — patterns (not in formalistic definition of pattern) of trying to find bugs in software. He mentioned Al Jorgenson’s dissertation where he identified a series of tasks that focus on various constraints of the application — input constraints, output constraints, device interaction constraints, and file system constraints (file system as a logical concept, not a device concept). James Whittaker has written a book (How to Break Software) on these attacks as well.

So, a large part of exploring for Cem is having a large batch of ideas of how to test software, and then determining which ideas to use. This is critical, and Cem thinks it’s the most difficult technical area of testing.

James suggested that to learn more about testing, one should read NTSB accident reports for examples of excellent testing and excellent thinking. He also recommended learning more about thinking scientifically. He also said that if you’re going to do exploratory testing really well, you need to have a model in your mind of what is a test — what are the parts of a test and how those parts can break down. He gave examples of coverage and oracles. For coverage, a tester will think of the areas they’ve covered, but they might not think of areas they hadn’t tested and have a wrong impression of what they’ve covered. Oracles are the method by which you recognize that a problem has occurred (generally, an oracle arrives at an answer via an independent method, though there can be other types of oracles as well). But, oracles can have problems too, and no oracle is going to be perfect and enable you to see every problem in the application.

The final element of exploratory testing that James talked about is humility. Even given intelligence, heuristics, and experience, people will make mistakes. Recognizing that and working to get differing viewpoints from others makes exploratory testers more effective.

(2011 update: Updated link for NTSB accident reports).