WHY,WHEN,WHAT AND HOW
Karen N. Johnson
WHO AM I?
• Karen N. Johnson is a longtime
contributor to the software testing
• Contributing author to the book,
"BeautifulTesting" by O’Reilly
• She has published numerous
articles,blogs and tweets about her
• Find her onTwitter as @karennjohnson
(note the two n’s)
• And her website: www.karennicolejohnson.com
• Karen was previously an independent
consultantbut is now employed at JAMF as a
Software Engineering Director.
“Visible” test automation doesn’t mean hosting test
automation demos - where people physically watch
“Visibility” is about creating transparency, knowledge and
ultimately helping people to understand what has been
built, what has not been built and the reasoning behind
OPEN THE DOOR TO COLLABORATION
• Developers can extend functions,offer code
reviews and tackle some of the more technical
aspects that might be out of reach for a test
• Manual testers can pick up where automation
• In some cases, other teams might be able to find an
opportunity to re-use what your team has built.All
possible from sharing.
BUILD TEAM OWNERSHIP
• When an automation test suite fails, people will care
more about resolving the issue if they know what
the automation suite contains and accomplishes.
• People will not engage with something they
• People will not own what they don’t know.
GAIN MANAGEMENT SUPPORT
Chances are there is a manager down the hall or
perhaps in a different country,and when budget
season rolls around as it always does, it’s best that
your manager has some understanding of what
you’re doing and the value the automation you’re
building brings to the team.
A sense of team ownership of the automation is
helpful as a team works towards deploying in a
CI/CD environment.This awareness increases the
possibility of other people on the team (other than
the test automation people) getting involved when
test automation fails.Think about it - why would
you care if automation fails,if you don’t know what
is lost when the automation is not working.
HOST ATESTAUTOMATION OPEN HOUSE
• Showcase your work.
• Automate based on a risk analysis and then
share that analysis.
• Explain the decision-making process.
• The remaining work is another way of indicating
the decision-making process that goes into
automation as well as possible limitations -
whether those limits are skills, tools or time.
• The beauty of making a recording is you have
the chance to say exactly what you want to say.
• A short video could provide
a multisensory explanation of the topic – which
is particularly useful for complicated subjects.
For example,you could be showing the testing
pipeline while you're explaining the same
information helping to show automation.
CREATE A QUARTERLY AUTOMATION UPDATE
Consider writing an email that provides a brief on
the state of automation.An email that is sent
frequently enough to be helpful without sending an
email so often,it is overlooked.An email update
gives you the opportunity to highlight the current
state of automation.
SHARE AT SPRINT REVIEWS
One benefit of sharing automation updates at the
sprint review is that the automation naturally
becomes seen and understood as part of the
product.Not as separate from the product.
• When a new feature is being pushed into
production use,knowing what has been automated
provides not only insight into what testing has
been done but some reassurance that if the
automation is added to a regression suite, issues
will be not only detected but potentially prevented.
• When automation lags product delivery (which
happens),it still helpful to know what has been
built and what work remains to be done.
• The regression suite is often the primary reason
automation is built,so a clear understanding of
the regression suite is essential to understanding
the automation effort.
• The importance of choosing what is included in
the regression suite gives tremendous insight
into the thinking behind the automation.
Providing visibility into automation shows how
testing is working towards the overall product
A test automation suite for a product is more than
the sum of feature testing and should be even
larger than a full regression suite. Somewhere in
the mix of automation scripts,there should be
tests designed to cover some of the“ilities” of
testing such as scalability,maintainability and a host
of other "ilities."These are tests that are not run
on a regular basis but are often the tests that
cannot be achieved outside of automation.These
tests are some of the most compelling reasons to
even have automation.
• Keep it simple.
• List the automation built.
• Show the alignment of product delivery and the
• Explain (high level) the process of running
automation in connection to shipping to production.
• Identify what has been built then let others ask for
• Nearly every stakeholder knows automation tools
have limitations; share what’s been built as well as
explaining (when it makes sense to) where the tool
may have imposed limitations on what could be
• Equally most stakeholders realize that test
automation people may have limitations themselves
as to what they are able to build, there’s nothing
wrong in sharing the limitations of what you’re able
• When a feature ships to production, there may be
more automation work to be done.It is perfectly
ok to let your stakeholders know what work
• In addition to feature automation, there will be
regression testing and possibly "ility" automation
tests that continually need maintenance. Sharing
what backlog of automation tasks remain is helpful
for stakeholders to understand the current state of
THE MAP ANALOGY
• Have you ever used a map that showed every road,
body of water and other terrain information making
the map unreadable?
• Detailing too low level of information about test
automation can generate the same result – over
information creating under-informed people.
• On the other hand,too little information equally
leaves people under-informed.
• Strike the balance.Find out what people want to hear
about to provide a “map of information.”
a physical map with
many roads and
markings to the
point where the map
then showed this
map as an example
of how filtering and
in some cases
makes a map more
readable and helpful.
The credit for this map
belongs to Rick Steves.
MAKING TEST AUTOMATIONVISIBLE
“ACTIONS AREVISIBLE,THOUGH MOTIVES ARE SECRET.”