Many organizations are using virtualization technology to increase the number of available test machines. However, this increase creates a new dilemma for testers. How can you confirm that all test systems are running properly and not showing signs of serious defects? Since many companies are reluctant to add additional staff members to simply “babysit” test environments, teams are forced to do more with less. Stephen Burlingame shares a solution to this problem that takes a proactive, rather than reactive, approach—in the form of a daily status report—to gather data, perform analysis, and generate results. This analysis of each test machine helps expose defects that, if ignored, could pose serious issues once the software is deployed to production. Take away an understanding of how such analysis is completed and a meaningful report compiled. See how critical data points are collected and presented efficiently. Leave with an understanding of how to create a similar early warning tool for your own organization and change the conversations testers are having.

1.
W14
Test
Techniques
5/4/16
15:00
The
Canary
in
the
Coal
Mine:
Create
an
Early
Warning
Tool
to
Help
Your
Testing
Presented
by:
Stephen
Burlingame
Navicure
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.stareast.techwell.com/

2.
Stephen
Burlingame
Navicure
Stephen
Burlingame’s
introduction
to
software
quality
assurance
occurred
during
his
university
senior
project
where
he
discovered
critical
defects
in
software
designed
to
diagnosis
breast
cancer
in
digitized
mammograms.
Since
graduation,
Stephen
has
worked
exclusively
in
the
quality
assurance
field.
In
2006,
he
joined
the
medical
claims
clearinghouse
Navicure,
Inc.,
where
he
is
director
of
quality
assurance,
responsible
for
overseeing
the
testing
for
an
application
that
processes
more
than
one
million
transactions
daily.
Living
in
the
Atlanta
metro
area
with
his
wife,
Julia,
Stephen
has
served
as
president
of
Atlanta
SPIN
(Software
and
Systems
Process
Improvement
Network).
Â

3. The Canary in the Coal Mine
Creating an Early Warning QA Tool to Help Your Testing
1

4. Slides written by
Stephen Burlingame
Director Of Quality Assurance at
Navicure Inc., a medical claims
clearinghouse offering additional
billing and analysis tools for
medical professionals.
Board member of Atlanta SPIN, a
non-profit organization dedicated
to promoting software process
improvement
2

5. Who’s familiar with vCAC from
VMWare?
3

6. vCAC from VMWare Example Interface
4

7. Created CentOS Installation with Custom Software Deployed
Now our QA team can request environments on as
needed basis
Software build is deployed and configured automatically
Near “push button” operation
5

8. Yea! We’ve created virtual
test environments! Our work
is completed and now we
can all go home!
In reality the hard work is not
complete
You’ve solved some difficult
problems but possibly created
new challenges
6

9. Environment Chaos!
Old Setup
CERTIMP
New Setup
CERTIMP
Virtual
1
Virtual
2
Virtual
N
Virtual
3
7

10. Managing and keeping
track of environments is
getting out of hand!
QA engineers spend more time
figuring out what systems they
have and the health those
systems than ever before.
More systems = more problems
8

11. Problem: How Can We Monitor Those
Systems and Not Spend More Time in
Maintenance than Testing?
9

12. Solution: Create a Daily Report
This report will highlight system health and basic
performance
This report will focus on displaying key performance
indicators in graphical form that is intuitive
This report will alert on known conditions that indicate
defects
10

13. “QA Health Checker”
11

14. “QA Health Checker”
Data Examples
Start Simple!
o Disk space alerts
o Indicate services
are up or down
o Look for
directories
present or
missing or when
last updated
12

15. “QA Health Checker”
Data Examples
Get More Complicated!
o Write SQL queries
that look for “bad”
data or missing data
o Look for errors in logs
or indicate that error
logs have content
o Look for scenarios
that indicate system
is experiencing
issues
13

16. “QA Health Checker”
Data Examples
Let’s Get Crazy!
o Create a uniform
CSS
o Implement basic
pictographic
principles
o Fit this report into
retrospective
discussions so you
improve the
metrics captured
14

17. 15

18. 16

19. 17

20. A More
Technical Look
Test systems run nightly cron jobs
to gather data and place it on
their local hard drive
Central reporting machine knows
which IP addresses to read from
and pulls the stats into a prepared
report
Report is emailed to a distribution
list early every weekday
18

21. A More
Technical Look
Used Python as the scripting
language
Created a deployment script and
documented the release process
Documented the report contents
and general architecture
Used Confluence as the
documentation portal
19

22. Changing the Conversation With
Employees
20

23. Old Conversation
QA Engineers didn’t always know what was going on or
how interpret data. They complained about “phantom
issues” and “non-reproducible bugs”. Used vague terms like
“slow”, “buggy”, or “weird”. Documentation was simply not
enough.
21

24. New Conversation
QA Engineers know right away the types of issues they are
having or about to have in the near future. Team leads are
keeping their focus on early signs of issues leading up to
release. Management is having pro-active conversation
with employees rather than a reactive dialog.
22

25. Closing Thought
Recommendation:
Take a slight risk with this report and experiment
Keep it adjusting and evolving!
23

26. Time For Questions
Contact Information:
Email Address – sburlingame@navicure.com
If in Atlanta, come to an Atlanta SPIN meeting!
24