Cem prepared this list to help us discuss the outcome of this workshop. We spent this morning discussing and refining it (with John moderating, which I mention so that he can get his Toastmaster’s credit 🙂 ). All of these things are in the context of testing on an agile project, where people typically shift among many roles. The “tester” might have a job title as programmer, tester, or something else, but when she wears her testing hat, this is what she’s up to. These extend and complement the points posted in an earlier entry (2011 update – when I moved the blog, I updated this link. I’m not 100% certain that the post it links to now is the one that was originally linked to, but I am certain enough to change the link.)

• Exploratory testing is “simultaneous learning about the product, designing tests, and executing tests”
• Agile test groups exist to provide test-related services that add value to the project.
  Agile test groups provide valuable services to programmers, customers, and other stakeholders who may vary from project to project.

• A core purpose of testing is to expose and investigate product and project risks. Achieving this typically requires application of a diversity of testing techniques.
  Skilled testers serve as the headlights of the project.
  Caution: Agile testing is more then free-form exploratory testing

• Collaboration among testers, programmers, and other stakeholders is more highly valued on agile projects than details of process, practices, or tools
• The agile tester’s objectives will vary across projects, and as the work evolves within projects.
  Examples:
  • Help programmers understand the program as they create it
  • Create change-detector suites to facilitate refactoring
  • Help project management understand the state of the project
  • Find bugs
  • Comply with regulations

  Each of these may dominate the thinking of the test team at different times on the same project

• There are noteworthy similarities in approach between XP and exploratory testing. Both:
  • Emphasize and cherish skill, integrity and individual’s best judgement
  • Break tasks into timeboxed chunks that are explored and handled by a task owner (with help). “Stories” and “Testing charters” are very similar concepts in practice
  • Limit expenditure on document creation to what is demonstrably needed
  • Confront the learning curve through structuring workflow to allow for change as we learn more
  • Emphasize ways to keep the brain continually engaged when the person is doing technical work on the project
• A distinguishing characteristic of the agile test team is their degree of support for the interests and desires of the programmers they collaborate with.
  Testers perform services to many different types of stakeholders but on the agile project, they increase their focus on services for programmers.

• Skilled agile testers protect programmers from information overload. They apply judgement to the questions:
  • What issues should I raise with the programmer?
  • What issues should I investigate soon, before raising a subset with the programmer?
  • What issues should I begin to monitor but investigate later?
• Two of the vital services provided by agile testers are:
  • Development of tests that help guide/coach the task (specification by example)
  • Development of test suites that expose problems caused by change, especially by refactoring (change detection suites)
• Within XP, the primary goal of the “acceptance test” is to facilitate understanding (communication):
  • Acceptance tests draw attention to the essence of a benefit, and guide the programmer in the design of the benefit-enabling code
  • Other tests might be better designed for regression(change detection) or as scenarios intended to explore a broad range of risks associated with this benefit in the context of the larger program
• Automation, ease and rapidity of use, and maintainability are essential attributes of tests designed for change detection.
  These attributes may be much less important for tests designed and used for other purposes.

• As agile testers develop trust of and insight into the practices (including tests) of programming teams, they abandon or slash their investment in tests that are redundant with programmers’ work or for other reasons unlikely to provide much value.
  This might free the agile tester from most boundary tests, for example, leaving them more time to explore hostile attacks, write experienced-user scenarios, translate stories into acceptance tests, or build better change detection suites.

• Skilled testing often involves extensive preparation and research. It is normal for this work to be an ongoing activity rather than something completed early in the project.
  Like everyone else on the team, testers know less at the soing development of risk liststart of the project than they will know at any later time.
  Examples:
  • Config testing including lab setup, identification of relevant combinations, ongoing research on compatibility risk
  • Development and use of complex applications of the software under development
  • Ongoing development of risk lists
  • Bursts of detailed analytical work, to build models or other bases for selecting among complex tests
  • Ongoing development of tools
• The prime purpose of a bug tracking process is to help the team get the right bugs fixed. An informal process that achieves this process is often good enough.
  Introducing formality in order to achieve other purposes, carries several costs. Cost-benefit analysis and evaluation of the plausibility of actually achieving those other benefits are appropriate.

• Much of the technology of agile testing has been created as free software. Whenever possible, we should find ways to contribute to the community repository.

  
  Bret added:

• Exploratory testing is a process that conforms to the principles of Agile Development
• Exploratory testing naturally complements XP

  
  James added:

• Skilled exploratory testing can be a powerful complement to unit testing and customer acceptance testing on agile projects

  
  Brian:

• XP is a bet and the bet is that generalism trumps specificism (People switch roles as opposed to specialists)

  
  Cem:

• When you advertise for a programming job, you are already advertising for a specialist – maybe not a specialist among programmers, but a specialist none the less

We want to stress that we’re not ready to make industry-wide grand pronouncements. We don’t have the experience yet to do that. These are hypotheses that some of us believe, but there was quite a bit of controversy over some of the points as well. Further discussion and experience is required.