Exploratory Testing, A Guide Towards Better Test Coverage.pdf
1. Exploratory Testing, A Guide Towards
Better Test Coverage
Exploratory Testing is a black box testing technique which lays out the freedom to
testers to think out of the box with a minimal dependency on pre-designed answers
and step on a process of investigation and discovery to improve the quality of
application under test. In layman terms, Exploratory testing is said to be a detailed
series of testing by exploring the platform, designing the test cases, executing those
immediately and maintaining the results. The thumb rule of exploratory testing is
: minimum planning and maximum execution. Exploratory testing is a crucial
aspect of software testing as this is not a script based test, it involves the art of
being creative and productive. Nowadays, this type of testing is preferable as this
doesn’t define a fixed set of methodologies which a tester has to follow, the tester is
completely free to fly on his own path to play with an application and identify the
potential bugs. This is actually a simultaneous process that carries out the activities
of test design and test execution without formally documenting the test cases and
test conditions.
Like every other testing technique, exploratory testing also has some pros and cons,
let’s have a quick look at those.
Pros and Cons of Exploratory Testing
2. Pros:
As this type of testing doesn’t require any extensive test planning, the testing can even be
done when requirement documents are not easily available.
It is more efficient in short term projects where minimal planning is required and important
bugs are found quickly.
This testing technique uncovers the maximum defects when compared with usual
regression testing as the regression testing is only performed according to a fixed set of
test cases.
This type of testing can also be done by the management team and stakeholders since
this doesn’t require any kind of scripting.
In an agile environment where continuous deployments are done, exploratory testing is a
perfect testing style that can share the insights in a limited timeframe.
Cons:
As exploratory testing is a crucial part of testing, it is more dependent on the tester’s prior
skillset, knowledge and experience.
As no formal documentation is made before performing exploratory testing, this might
create a negative impact in long-term projects.
3. It is difficult to review the test cases later which creates a possibility of missing the critical
bugs.
Techniques of Exploratory Testing
Strategy Based Exploratory Testing
This type of exploratory testing is mostly done by testers that are a bit familiar with an
application’s functionalities that need testing. The familiarity is important to define the
testing strategies. The strategy to test is usually developed using various analysis such as
Boundary Value analysis, equivalence technique and risk-based technique to identify
more potential bugs.
Freestyle Exploratory Testing
As the name suggests, this technique provides a free style to the testers to investigate
bugs or defects without any detailed planning. This approach is basically considered as
the combination of smoke testing and monkey testing. This technique is usually carried
out by a tester when there is a need to get familiarized with an application and to perform
parallel continuous testing without defining any ground rules for test coverage.
Scenario Based Exploratory Testing
This type of testing is usually carried out after the initial phase of testing to make a
variation in testing flow as per the application learning and observations from the initial
testing process. The idea is to develop end to end scenarios according to the real user
interaction. Testers tend to prosper different sets of possibilities by exploring the platform
to match each developed scenario and provide maximum test coverage.
4. Skills Required for Exploratory Testing:
Exploratory testing is a dynamic and creative testing approach that relies heavily on
the skills and mindset of the tester. To effectively conduct exploratory testing, testers
should possess the following skills:
Critical Thinking: Exploratory testing requires the ability to analyze and evaluate the
software under test from different perspectives. Testers need to think critically to
identify potential risks, uncover hidden defects, and make informed decisions on
where to focus their testing efforts.
Creativity: Testers should be able to think outside the box and come up with
innovative testing ideas. They need to explore different scenarios and test paths that
may not be covered by scripted test cases. Creative thinking helps testers uncover
unique defects and ensure comprehensive test coverage.
User Perspective: Exploratory testers must have the ability to think like end users.
They should consider how users interact with the software, their expectations, and
the potential use cases. This user-centric mindset helps testers identify usability
issues, evaluate the software’s intuitiveness, and ensure a positive user experience.
Communication Skills: Effective communication is crucial for exploratory testers.
They need to collaborate with development teams, product owners, and other
stakeholders to gather information, share insights, and report defects. Clear and
concise communication helps convey findings accurately and ensures efficient
collaboration within the testing team.
Domain Knowledge: Testers should have a solid understanding of the domain in
which the software operates. This knowledge enables them to ask relevant
questions, anticipate potential risks, and design test scenarios that align with the
5. domain-specific requirements. Domain knowledge helps testers uncover defects
specific to the industry or application context.
Analytical Skills: Exploratory testing requires testers to analyze complex systems
and identify patterns or anomalies. Strong analytical skills help testers make sense
of data, interpret system behavior, and identify potential areas of concern. They
should be able to draw conclusions based on observations and make informed
decisions about further testing.
Attention to Detail: Testers should possess a keen eye for detail to catch even the
smallest defects. They need to observe system behavior, user interfaces, and data
flows meticulously, ensuring that no issues go unnoticed. Attention to detail helps in
identifying subtle defects and ensuring a high level of software quality.
When and How to use Exploratory Testing in testing workflow
Exploratory Testing is said to be done in early phases of Software Development Life
Cycle(SDLC). In the agile development environment where the sprints are short and
software builds are released very frequently, exploratory testing plays a vital role in
discovering and reporting bugs and getting them resolved within the same sprint
cycle. The experience gained by testers from exploratory testing can be valuable in
later stages of testing and to design in-depth test cases. Once the test cases are
designed from such investigation and observation, they can further be planned to be
automated to get added in the regression suite.
Till now we are aware that exploratory testing is all about exploring. Exploratory
testing is not only done to understand the product functionalities but also to
understand the customer requirements. For quality bench marking, it is important to
6. navigate the software according to the end user perspective. Though there is no
defined process for exploratory testing, the below depiction will define a structure of
exploratory testing commonly adopted by testers.
Real Time Example of Exploratory Testing
Example #1
What we see is what we believe, right? You might still be wondering how exploratory testing
looks like. Well, it looks like wandering out in an app and choosing your own path to fly
without depending on someone’s direction. That means, there are no pre-designed test
cases and testers have some sort of quick description which defines what exactly needs to
be tested.
Let’s take an example of a food delivery app and list few testing modules that strikes in our
mind:
Login
Search and filter
7. Outlets nearby
Restaurant selection
Adding food items to cart
Modifying cart
Promo codes
Payment Gateway
Delivery tracking
In case of product exploratory testing, it is a good practise to start the testing from
the initial module like login page and then move on to the next relevant module.
Following this testing workflow would also cover the testing according to the end
user perspective. Always remember, speed and accuracy are the most important
factors in such applications.
Since the food delivery app would be having multiple restaurants and dishes, it is
important for an exploratory tester to test with different test data. The login page and
payment gateway is a more concerned aspect in terms of security and is important to
be tested with multiple test scenarios. However, being creative and analytical is
something that exploratory testing requires the most.
Example #2
Let’s take another example of exploratory testing using pCloudy Certifaya AI-Bot.
The new trend of intelligent automation has proposed the new way of exploratory testing.
The concept of AI Testbot has been introduced to reduce the burden on the QA team.
In exploratory testing, AI Testbot investigates and reports bugs in case of functionality
crashes or unexpected errors pop-ups. For each test iteration, it has a power to store
execution logs, video recordings and screenshots for future references.
8. pCloudy’s Certifaya features a testbot that allows a tester to upload an application
that needs to be tested and sit back tight until the test report is handed over. The key
feature of this testbot is that it not only performs exploratory testing but also the
cross device testing on different device variants. Once an application is uploaded
and a session is triggered, the smart bot crawls inside the application without any
human intervention and digs for each random corner case with a motive to uncover
maximum bugs. As soon as the Certifaya session gets completed, an insightful
report gets mailed to the team to document the test results.
Certifaya Report
The comprehensive report includes scenarios performed by the testbot along with
logs and snapshots. The functionality performance score is also wrapped up in the
9. report in the form of charts and graphs like battery chart, memory chart, CPU chart in
addition to each frame rendering time.
In agile methodology where software is released in multiple small iterations and
developers have to wait for rapid feedback from testers, the testbot plays a vital role
in performing quick exploratory testing for new software versions and smoke testing
for the entire build deployed.
Myths about Exploratory Testing
Let us discuss about some misconception related to exploratory testing that mislead
a tester in his way to become an expert:
1. Myth: Exploratory testing doesn’t require planning and documentation
Many people think that exploratory testing can be done without even minimal planning and
without any sort of documentation.
Reality: This is a big myth that the testers need to incinerate before starting off with
exploratory testing. It is a structured approach of testing, the way of structuring
completely depends on a tester which definitely needs some kind of planning to
make a move. Since the probability of change in requirements is so high in agile
development, without documenting the testing plans and statuses, exploratory
testing cannot be possible.
2. Myth: Exploratory testing is similar to Ad-hoc testing
Though some of the factors of exploratory testing match with ad-hoc testing, that doesn’t
mean that these two approaches are completely similar.
Reality: Ad-hoc testing is a very informal and random approach of testing that
doesn’t specify any strategy towards testing. In ad-hoc testing, the agenda is to first
learn the background processes of an application and then perform the testing
10. whereas in exploratory testing, the agenda is to understand an application while
exploring and testing.
3. Myth: Exploratory testing is not applicable on complex systems
Sometimes the complexity of the system enforces testers to think that exploratory testing
cannot be applied to a complex system which requires in-depth planning and training.
Reality: Exploratory testing gives testers a freedom to choose their own way to test,
this helps testers to understand these complex systems more while crawling in-
depth. The more complex an application is, the more test cases get designed and
implemented which lead to a better coverage of the system.
4. Myth: Either exploratory testing or scripted testing can be done
Due to the short span of time in sprint, it is obvious for such myths to come around,
however, the myths should be clear before it is too late.
Reality: 100% coverage with scripted testing just cannot be possible. Though the
implementation of scripted testing requires a lot of prior planning and designing of
test scenarios and test cases, exploratory testing cannot be replaced with scripted
testing. In exploratory testing, the tester gets a chance to carry out additional tests
that were never pre-defined. In the process of exploratory testing, the critical test
cases can be dug out to make sure they can be scripted later on for better test
coverage.
Achieving success in exploratory testing with test automation
Many people have a question in mind, can exploratory testing be automated?
11. We can achieve success in testing with a combination of exploratory testing
and test automation, however, these two approaches cannot replace each
other.
After being aware of the exploratory testing approach, do we still think that sense of
creativity be automated? Can human curiosity be automated? Or random inventions
automated? I don’t think so.
Though these kinds of activities can be partially done by a testbot but it is actually
not possible via test automation. Common things that we can automate in flow of
exploratory testing is to generate random test data, functional test execution, output
logging, developing reports and sharing insights with the development team.
In an agile development model, the process structure is defined in such a way that
the testing can be done parallely with development. Once the new build is deployed
on a testing environment, testers get a go ahead from dev to start off the testing.
To start exploratory testing, initial minimal plans are developed and testing is
continued accordingly. Each edge case, critical case, flaky test case and other
anonymous doubtable test cases are recorded separately so that they can be
retested.
It is very complex and time consuming for the QA team to retest such test cases
everytime a new build is deployed in a testing environment. Hence, once the
exploratory testing of a particular module comes to an end, a new plan is developed
to automate such test cases which can have direct impact on the performance of the
production environment. In this way, we can achieve success in exploratory testing
12. with test automation which further leads to better test coverage by covering all corner
cases and uncovering all major defects and serving continuous improvement in
quality.
Common Challenges and Solutions in Exploratory Testing:
Exploratory testing, like any testing approach, comes with its own set of challenges.
Being aware of these challenges and having strategies to overcome them can
significantly enhance the effectiveness and efficiency of exploratory testing. Here are
some common challenges and their corresponding solutions:
Lack of Test Coverage: One challenge in exploratory testing is ensuring sufficient
test coverage. Since there are no predefined test cases, there’s a possibility of
overlooking certain areas of the software. To address this, testers can create test
charters or test missions that outline the areas to be explored during the testing
session. These charters act as a guide and help ensure comprehensive coverage.
Limited Time and Resources: Time constraints and resource limitations can
impact the extent of exploratory testing. Testers may feel pressured to rush
through testing, compromising the thoroughness of their exploration. To overcome
this challenge, prioritization is key. Testers should prioritize testing based on risk,
critical functionalities, or areas prone to defects. They can also leverage
techniques like session-based testing to allocate specific time blocks for focused
exploration.
Documentation and Reporting: Exploratory testing typically lacks formal
documentation, which can make it challenging to track and communicate findings.
Testers should adopt lightweight documentation approaches, such as capturing
13. notes, screenshots, or video recordings during testing sessions. It’s essential to
document important observations, test ideas, and defects for future reference and
reporting to stakeholders.
Adapting to Agile Development: Exploratory testing is well-suited for agile
development environments, but it can be challenging to align with fast-paced
iterations. Testers must be adaptable and flexible, ready to explore new features
or changes rapidly. Testers can collaborate closely with developers and product
owners, participate in agile ceremonies, and utilize tools that support quick
feedback and communication.
Subjectivity in Testing: Exploratory testing involves subjective decision-making
by testers, which can lead to variations in approaches and coverage. To address
this challenge, establishing guidelines and heuristics can provide some structure
while maintaining the flexibility of exploratory testing. Testers can define common
testing patterns, share best practices, and engage in team discussions to ensure
consistency and alignment within the testing team.
Skill and Knowledge Gaps: Exploratory testing heavily relies on testers’ skills
and experience. It can be challenging for less-experienced testers to effectively
explore the software. To bridge skill gaps, organizations can invest in training
programs or mentorship to develop the necessary testing skills. Pairing less-
experienced testers with more experienced ones can also help transfer knowledge
and improve the effectiveness of exploratory testing.
Defect Reproducibility: One challenge in exploratory testing is the ability to
reproduce defects reliably. Testers may encounter intermittent issues or struggle
14. to recreate specific conditions leading to defects. To address this, testers should
capture detailed steps, input data, and environmental conditions during testing.
Clear and precise defect descriptions help developers reproduce and fix issues
more effectively.
Managing Test Data: Generating and managing diverse test data can be
challenging in exploratory testing. Testers need to ensure they have relevant and
representative data for various scenarios. Using data generation tools and
techniques can assist in creating different data sets quickly. Testers can also
collaborate with developers to set up test data environments or leverage
techniques like data masking to anonymize sensitive data.
Collaboration and Communication: Effective collaboration and communication
among testers, developers, and stakeholders are crucial in exploratory testing.
Lack of communication can lead to misunderstandings or delays in addressing
issues. Testers should actively engage in discussions, provide timely feedback,
and participate in cross-functional meetings. Utilizing collaboration tools and
maintaining transparent communication channels can facilitate efficient teamwork.
By being aware of these challenges and implementing appropriate solutions, testers
can overcome hurdles and conduct more effective and comprehensive exploratory
testing. Continuous learning, adaptation, and leveraging best practices contribute to
the success of exploratory testing efforts.
Wrapping Up
Without any doubt, exploratory testing brings out the creativity of a tester and has an
extreme effect in improving the product quality. Investing too much time in exploring
and discovering edge case scenarios might affect the regular testing, getting a AI bot
like Certifaya to do some of the testing is definitely helpful. Also, It is important to
15. maintain a healthy balance between the two. The kind of reporting done in
exploratory testing matters a lot. The tester may find out only one bug in exploratory
testing but the way the reporting is done can change the decision made by the
management to enhance the growth of the product.