Making exploratory testing effort visible and reproducible by tracking your testing notes in version control.

1. @theBConnolly
brendanconnolly.net
Brendan Connolly
Carpe Data
AS CODE

2. SLIDESMANIA.COM
A SHIFT LEFT
SUCCESS STORY
Looking at how Infrastructure as Code has
enabled the DevOps revolution
@theBConnolly

3. SLIDESMANIA.COM
TERMS & DEFINITIONS
Shift Left
Taking a task that's traditionally done at a later stage of the
process and perform that task at earlier stages
Infrastructure as Code (IaC)
The managing and provisioning of infrastructure through code
instead of through manual processes.
https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac
https://devopedia.org/shift-left
@theBConnolly

4. SLIDESMANIA.COM
WHAT SHIFTED?
Understanding what success looks like
@theBConnolly

5. Develop
Design / Implement
Release
Deploy / Monitor
Test
Validate / Confirm >
<
<
@theBConnolly

6. Develop
Design / Implement
Test
Validate / Confirm >
<
@theBConnolly

7. Develop
Design / Implement
<
Plan
Do
Review
@theBConnolly

8. Review
Typical
Development
Tasks
Deploy & Monitoring
Tasks 🧈🍴🍞
<
Do
Plan
@theBConnolly

9. SLIDESMANIA.COM
MODES OF TEST
MANAGEMENT
How we manage testing efforts
@theBConnolly

10. SLIDESMANIA.COM
TEST CASES
AS A PRACTICE
Quick overview of managing testing efforts
using test cases
@theBConnolly

11. dgqsdvas
@theBConnolly
Testing requires creating a direct relationship
between requirements and demonstration of
successful completion of said requirement.
And the basic unit for that is the test case.
Understanding Test Case Management, SmartBear
“

12. SLIDESMANIA.COM
TERMS & DEFINITIONS
Test
A set of one or more test cases.
Test Case
A set of preconditions, inputs, actions
(where applicable), expected results and
postconditions, developed based on test
conditions.
Test Step
A single interaction between an actor and
the test object consisting of an input, an
action, and an expected result.
Test Run
The execution of a test suite on a specific
version of the test object.
Test Suite
A set of test scripts or test procedures to
be executed in a specific test run.
https://glossary.istqb.org
@theBConnolly

13. SLIDESMANIA.COM
TEST CASES
Centralization
Testing knowledge, status and reporting all in 1 place
Visibility, Traceability & Coverage
Auditable history and reporting of test activities
Productivity & Efficiency
Simplified Status, Shareable steps, Workload flexibility
Value Proposition
@theBConnolly

14. SLIDESMANIA.COM
EXPLORATORY
TESTING
AS A PRACTICE
Quick overview of managing testing efforts
using Exploratory Testing
@theBConnolly

15. dgqsdvas
@theBConnolly
Exploratory testing values human insights and
intuition that you can’t reach with machines,
automation and scripted testing.
Solutions for Exploratory Test, Xray
“

16. SLIDESMANIA.COM
TERMS & DEFINITIONS
Exploratory Testing
An approach to testing whereby the testers
dynamically design and execute tests
based on their knowledge, exploration of
the test item and the results of previous
tests.
Ad Hoc Testing
Informal testing performed without test
analysis and test design.
Test Charter
Documentation of the goal or objective for a
test session.
Test Session
An uninterrupted period of time spent in
executing tests.
Session-Based Testing
An approach in which test activities are
planned as test sessions.
https://glossary.istqb.org
@theBConnolly

17. SLIDESMANIA.COM
EXPLORATORY TESTING
Uncover the Unknowns
You can’t predict all potential failures that could happen without
deeply exploring your system.
Human Insights and Intuition
The human knowledge and experience coupled with technical
skills and understanding of the systems uncovers more bugs
Value Proposition
https://www.getxray.app/solutions/exploratory-testing
@theBConnolly

18. SLIDESMANIA.COM
EXPLORATORY
TESTING
AS CODE
Exploratory Test Management inspired by
IaC
@theBConnolly

19. dgqsdvas
@theBConnolly
Poetry is what is lost in translation. It is also what is
lost in interpretation
Robert Frost
“

20. SLIDESMANIA.COM
IN PRACTICE
Utilize dev tooling to satisfy the value
proposition of Test Cases while
maintaining the value of Exploratory
Testing to improve testers skills and quality
of life
@theBConnolly

21. SLIDESMANIA.COM
THE STACK
The tools and tech needed to implement
Exploratory Testing as Code
@theBConnolly

22. SLIDESMANIA.COM
VERSION CONTROL
Enables teams to collaborate and
streamline development to resolve
conflicts and create a centralized location
for code.
https://about.gitlab.com/topics/version-control/
@theBConnolly

23. SLIDESMANIA.COM
VERSION CONTROL
Why use it for anything besides code?
● Git Repository acts as the single source of truth
● Collaborate in change requests using code
reviews, comments, & suggestions.
● Commits to the main branch act as an audit log
@theBConnolly

24. SLIDESMANIA.COM
MARKDOWN
A plain text formatting syntax used to
write content on the web.
Commonly used by writers & programmers
to write quickly without having to take
time using the formatting toolbar of text
editors.
@theBConnolly

25. SLIDESMANIA.COM
Where is it used?
● Readme files
● Blog Posts
● Web App View
Templates
● Merge/ Pull Requests &
Issues
● Slack Messages
● Jira / Confluence
MARKDOWN
@theBConnolly

26. SLIDESMANIA.COM
THE PROCESS
@theBConnolly

27. SLIDESMANIA.COM
START A BRANCH
Create Feature Branches
● Each new ticket in its own branch
● Only Complete, Working tickets get merged
● Main branch should represent what is live in
production
@theBConnolly

28. SLIDESMANIA.COM
TAKE NOTES
Show Your Work
● Document what you actually do
● Include relevant context
● Avoid excessive detail
@theBConnolly

29. SLIDESMANIA.COM
TAKE NOTES
Support with Evidence
● A Picture is worth a 1000 words
○ Links, Logs, Data… too
● Use your best judgement
○ A little goes a long way
○ Remember your reader
@theBConnolly

30. SLIDESMANIA.COM
TAKE NOTES
Organization
● Directory for work item being tested
● 1 document per session
○ Additional follow up sessions go in a separate file
● Use a data subdirectory for related screenshots etc.
@theBConnolly

31. Notes
Sample Directory
Structure
> tree
.
|___TKT-123
|___session.01.md
|___data
|___example.jpg
|___export.csv
@theBConnolly

32. Notes
Sample Session
Markdown
> cat ./TKT-123/session.01.md
# TKT-123 User Export Flow
Ticket description here
—--
## Notes
Began by loading the user selection form …
…
Selected users for export with …
[sample export data](./data/export.csv)
…
—--
## Follow Ups
- …
@theBConnolly

33. SLIDESMANIA.COM
PEER REVIEW
Code reviews, also known as peer reviews,
act as quality assurance of the code base.
Code reviews are methodical assessments of code
designed to identify bugs, increase code quality, and
help developers learn the source code.
https://about.gitlab.com/topics/version-control/what-is-code-review/
@theBConnolly

34. SLIDESMANIA.COM
Benefits of Code Review
● Discover Bugs Earlier
● Maintain Compliance
● Enhance Security
● Share Knowledge
● Increase Collaboration
● Improve Code Quality
PEER REVIEW
https://about.gitlab.com/topics/version-control/what-is-code-review/
@theBConnolly

35. SLIDESMANIA.COM
PEER REVIEW
Share Knowledge: Helps team members take vacation,
team has background knowledge on a topic
Increase Collaboration: Work together to create a
solution, increasing ownership of their work.
Improve Quality: A human who knows your code base can
notice code quality issues that automated tests may miss.
https://about.gitlab.com/topics/version-control/what-is-code-review/
@theBConnolly

36. SLIDESMANIA.COM
MERGE
Commit Information into a Shared Testing
Knowledge Base
● Update Status: Merge marks a portion of work as done
● Publish Results: Merge integrates your effort into
collective testing history
@theBConnolly

37. SLIDESMANIA.COM
ITERATE
Start Simple, Start with what you know
● Time box
● Discover
● Reflect
● Repeat
@theBConnolly

38. Session 2
Follow ups
Bug Investigations
Session 1
Survey Initial Quality
Session 3
Bug Closure
Final Spot Check
@theBConnolly

39. Session 2
Follow ups
Bug Investigations
Session 1
Survey Initial Quality
Session 3
Bug Closure
Final Spot Check
> tree
.
|___TKT-123
|___session.01.md
|___session.02.md
|___session.03.md
|___data
|___unexpectedPrompt.jpg
|___start.jpg
|___more.as.needed.jpg
@theBConnolly

40. SLIDESMANIA.COM
ITERATE
Codebases get refactored as they grow &
evolve
Start with:
● Simple Branching
● Simple Naming
● Simple Template
Learn & Experiment
@theBConnolly

41. SLIDESMANIA.COM
OUTCOMES
@theBConnolly

42. SLIDESMANIA.COM
MEASURING UP
Centralization
Testing knowledge, status and reporting all in 1 place
Visibility, Traceability & Coverage
Auditable history and reporting of test activities
Productivity & Efficiency
Simplified Status, Shareable steps, Workload flexibility
Test Cases Value
@theBConnolly

43. SLIDESMANIA.COM
MEASURING UP
Centralization ✅
Testing knowledge, status and reporting all in 1 place
Visibility, Traceability & Coverage ✅
Auditable history and reporting of test activities
Productivity & Efficiency ✅
Simplified Status, Shareable steps, Workload flexibility
Exploratory Testing as Code Value
@theBConnolly

44. SLIDESMANIA.COM
AGILE ALIGNMENT
What is a Spike?
A time-boxed research activity that helps teams make
better decisions & deliver better products
https://www.mountaingoatsoftware.com/blog/spikes
@theBConnolly

45. SLIDESMANIA.COM
AGILE ALIGNMENT
An Implicit Quality Spike on every ticket
Spike process typically includes a report,
status update, document demonstrating the
process and outcomes of the effort.
@theBConnolly

46. SLIDESMANIA.COM
SHARED ECOSYSTEM
Tooling Experience Value
Version Control, Peer Review, CI/CD workflow experience builds
higher value skills helping advance testers careers.
Increased Collaboration
Shared tools gives more surface area for different roles to learn
from and support each other.
Discoverability
Developers more to stumble across testing efforts and get
involved
@theBConnolly

47. SLIDESMANIA.COM
ASYNC/REMOTE
FRIENDLY
Async Out of the Box
Barrier free collaboration using tools designed for
distributed asynchronous communication.
Minimally Invasive Mentorship
High transparency of work opens a window of
opportunity to enable, train and grow testers skills.
@theBConnolly

48. SLIDESMANIA.COM
SCALABLE
Better Onboarding
Working the process naturally supports integrating new testers.
Better Mentorship
Effectively coach “meta” skills
Better Knowledge Transfer
Passively accumulate a repository of actual testing including
not just summary data but full searchable, referenceable
history.
@theBConnolly

49. Questions?
@theBConnolly
github.com/brendanconnolly/etasc_eurosta
r23.