The document outlines the principles and practices of exploratory testing (ET), emphasizing its structured yet flexible approach that allows for concurrent planning, design, and execution. It defines the characteristics of ET, distinguishes it from scripted testing, and offers guidance on initiating and expanding ET practices within organizations, including measurement and continuous improvement. Additionally, the document highlights the benefits of ET, such as increased engagement, faster feedback, and a better alignment with customer needs.

1. Start, Grow, and Perfect Exploratory Testing on Your Team
Kevin Dunne
Director of Product Strategy
QASymphony
Kimberly Stockett
QA Manager
KMS Technology

2. Agenda
1. What is exploratory testing (ET)?
1. What is NOT ET?
1. How can I start an ET practice?
1. How can I expand the footprint of my ET practice?
1. How can I measure & improve the effectiveness of my ET practice?

3. What is exploratory testing?
1. Parallel test planning, test design, and test execution
1. Specific yet flexible
1. Aligned towards investigation of potential opportunities
1. Values depth and attention to detail during testing
1. Fosters knowledge sharing and awareness

4. Parallel Planning, Design & Execution
Unlike traditional testing techniques, planning, design, and execution happen
concurrently, allowing efficiencies of time as well as flexibility in approach
Plan Design Execute Report
Plan
DesignExecute
Report

5. Specific yet Flexible
Exploratory testing provides a specific lens through which to perform testing –
whether that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily
as the product owner envisioned it
Manual Scripted Testing
I tested the application as the script prescribed
Exploratory Testing
I tested the application as the end user would

6. Investigating Opportunities
Exploratory testing rewards testers who identify unknown areas of “opportunity”
within the application, as they are essential in maintaining a backlog of future
test charters
Manual Scripted Testing Exploratory Testing

7. Knowledge Sharing
Exploratory testing relies on knowledge sharing to reach full potential –
developing testers who understand the impact of more areas of the application
allows them to identify more areas of risk and opportunity
Plan
DesignExecute
Report
Transfer
Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?

8. What is Not Exploratory Testing
1. Exploratory testing is NOT unstructured testing
1. Exploratory testing is NOT the only form of testing
1. Exploratory testing is NOT throwaway work
1. Exploratory testing is NOT impossible in a regulated
environment

9. ET is NOT Unstructured Testing
While exploratory testing allows for flexibility in the exact path of the application that is
tested, it is NOT unstructured, in that it still contains parameters such as:
1. A goal of the exploration
1. A log of the activity performed
1. A lens through which the testing is performed (i.e. a user persona)
Performing exploratory testing without involving some parameters such as the above
allows a greater risk of unsuccessful implementation of exploratory testing

10. ET is NOT the Only Form of Testing
Exploratory testing is best suited as a complement to automated and manual scripted
test cases. It can feed these types of testing to create greater depth in testing, and also
to identify any potential gaps in coverage.
Potential New Feature Testing Cycle
Code
Developer
Unit Test
Exploratory
Testing
Manual
Scripted
Test
Automation
Regression
Test
Exploratory
Testing
Feature “Delivered”
“Let’s make sure this
is worth writing
scripts against yet”
“Let’s make sure
were still testing all
aspects of this”

11. ET is NOT Throwaway Work
Exploratory testing does NOT need to be extra work done on top of other testing
methods – it can count on its own towards testing progress and coverage if properly
accounted for. Some of the necessary information needed to manage it:
1. Charter
2. Session Sheet
3. Oral report
4. Debrief
5. Data Files
6. Logs
"Any testing approach is manageable when you choose to manage it.” – Michael Bolton
http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/

12. ET is NOT Impossible in a Regulated Environment
Contrary to rumor and popular belief, exploratory testing is not only allowed in most
regulated environments, it is also essential.

13. Starting an Exploratory Testing Practice
1. Who - choosing the pilot group
1. What - choosing the application
1. Where - choosing the team/location
1. When - choosing the release cycle
1. How - choosing the Exploratory Testing framework/approach

14. Who will be the first to explore?
Finding the right group of initial members to pilot exploratory testing is one of the most
critical aspects of successful implementation. Pilot members should be:
• Comfortable with uncertainty and flexible
• Well connected in the organization
• A group that covers the ranges of skill and experience on the team
• Solid communicators and willing to speak up about their concerns
• Attuned towards new experiences as well as personal and team improvement

15. What will be the first application we explore?
The pilot application should be one that minimizes the risk to the business of serious
impact if exploratory testing is unsuccessful. Some factors to look out for in a pilot
application are:
Size
Small to medium sized
application – easier to
track coverage
Time on Market
Mature application –
easier to see net impact
of explorations
Business Criticality
Not the flagship product –
executives will allow a
longer period to pilot

16. Where will the first exploratory team be located?
To ensure the highest likelihood of success, it is important to choose a team that is
located centrally with easy access to developers and visibility to executives overseeing
the implementation
Centrally Located
Allow for collaboration
and knowledge sharing
early on
Access to Developers
When developers are
near by, testers can
provide feedback quickly
Executive Visibility
Locating the team near
execs means better
visibility of impact

17. When will we first introduce exploratory testing?
It is critical to introduce exploratory testing during a time where the pilot group can
contribute their full attention to the new initiative and they are not being pulled in other
directions:
• Avoid “busy season” – typically around big end of year releases
• Avoid releases/sprints that overlap with long holidays
• Avoid times of the year when application usage is high (i.e. year end closing for an
accounting application)
Be cautious of implementing exploratory testing while other change initiatives are going
on. It can be helpful as there is a change “tailwind”, but be aware exploratory testing
might be falsely attributed to other initiatives’ failure

18. How will we structure our exploratory testing?
Introducing exploratory testing within a framework will greatly increase the odds of
success, and will reduce fear and uncertainty among the practitioners as well as
executives.
Session Based Test Management is a popular framework for this, as it tracks all the
important data on testing:
More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf
• Session charter (includes a mission statement, and areas to be tested)
• Testers involved Date and time executed
• Task breakdown
• Data files
• Test notes
• Issues
• Bugs

19. Expanding the Footprint
1. Build champions within the organization
1. Share best practices and tips
1. Understand the limits and opportunities

20. Team #3 Team #3 Team #3
Build Champions within the Organization
There are two main ways we can build champions to spread the word
about exploratory testing in our organization:
Initial
Team
New
Team
New
Team
“Bowling Pin” Model “Infiltration” Model
Initial
Team
Tester
#1
Tester
#2
Tester
#3

21. Share Best Practices and Tips
A common threat in Agile organizations is the “information hoarding.”
As loyalty to the Agile team grows, teams can lose site of the importance of
improving the organization as a whole, and strive instead to improve there
position as the best team.
Ways to mitigate against this include:
• Greater alignment with organizational goals
• Transfer team members periodically
• Reward teams for sharing knowledge
• Provide greater visibility across teams

22. Understand the Limits and Opportunities
There is a good chance that you will find early success with exploratory
testing.
However, it is important to temper your excitement to roll out
exploratory testing across your organization.
Executives often have an incredible memory when it comes to failed
initiatives. Often, it is better to follow a measured approach than risk
having exploratory testing barred from your organization entirely.

23. Measure and Improve ET Practices
1. What are we trying to accomplish by implementing ET?
1. How will we measure ET’s ability to accomplish these
goals?
1. How will accomplishing these goals improve our ability
to expand ET in the organization?

24. Benefits of Exploratory Testing
1. Increases focus on unique scenarios with greater potential to find new
bugs
1. Drives engagement from testers and fosters critical thinking during the
testing process
1. Boosts team morale and provides a change of pace from manual scripted
testing
1. Encourages testers to think as end users and the "voice of the customer"
and drive improvement in the product

25. SMART Goals
• Specific
• Measurable
• Attainable
• Relevant
• Time-bound

26. Making the Case for Exploratory Testing
• Claim: ET will create more educated testers
• Metric - increase in average # of testers capable of test a given
scenario
• Claim: ET will provide faster feedback on new features
• Metric - reduction in user stories revised in subsequent sprints
• Claim: ET will increase the alignment with customer needs
• Metric - reduction in support tickets/feature requests
• Metric - increase in net promoter score/referenceable customers
• Claim: ET will garner communication product, dev, and testing
• Metric - reduction in open defect time
• Metric - increase in acceptance criteria met

27. Continuous Improvement
The path to success with exploratory testing does NOT end when implementation is
complete.
You must continually improve to make sure that your program continues to add value in
the organization
Plan
Implement
Evaluate
Reassess
Continuous Improvement

28. KMS has successfully implemented Exploratory testing techniques for all clients seeing benefit in quality and
efficiency to the act of testing with an added bonus of boosted morale for the QA Analysts.
• Benefits we have seen
• Time boxed testing creates structure and
limits
• Risk is apparent from application reviews
• Faster and deeper product understanding
with hands on exploration
• Proactive over reactive approach in
discovering bugs or workflow issues
• Items to keep in mind
• Worm holes in testing – knowing when to end a
testing path is based on expereince of the QA
Analyst
• Experiences drives when to annotate in the tool
(QASymphony eXplorer) to make defect
referencing easier
• Session definition – too much or too little
KMS is a fast growing global service provider with over 500 professionals trained on QA Symphony
products. We are a Platinum Partner with QASymphony with over 3 years experience in Exploratory
Testing Techniques.
About KMS Application of ET

29. Efficiency
In a month long project consisting
of 4 1 week sprints the team
uncovered 146 bugs in 143 hours
averaging 1.02 bug/hour
Quality
The team completed Mind Maps outlining
154 Charters completed 144 charters in
143 true testing session hours identified
high priority bugs
Dramatic & Measurable Results from ET
Value (Bugs Found)
20 High priority at 14%
57 Medium Priority at 39%
66 Low Priority at 45%
3 Closed as N/A at 2%

30. Resources/Thought Leaders
• James Bach - http://www.satisfice.com/
• Jonathan Bach - https://jonbox.wordpress.com/
• Michael Bolton - http://www.developsense.com/
• Paul Holland -
http://testingthoughts.com/blog/author/testthought
• Keith Klain - http://qualityremarks.com/
• Brian Osman - https://bjosman.wordpress.com/

31. Questions?
Kevin Dunne
kevindunne@qasymphony.com
Twitter: @kevindunneQA
Linkedin: www.linkedin.com/in/kevindunneQA
Blog: http://www.qasymphony.com/blog/
Kimberly Stockett
kimberlystockett@kms-technology.com