A solid test automation implementation is key to any foray into continuous delivery. Although the test automation waters may look peaceful and pristine, anyone who has jumped in to automate testing complex systems has found the waters to be troubled and murky. Tests take too long to run, they periodically break for mysterious reasons, or they fail to catch business critical problems. Over time this leads to automated tests becoming outdated and irrelevant. Brian Saylor has studied a number of test automation projects—and far too many lines of test code—to identify the fundamental concepts that are frequently missing. The test automation quagmire can be conquered by applying specific concepts. Brian uses a complex large scale website as an example and walks through identifying automated test goals, what tests should be automated and more importantly which should not, how to prioritize a test backlog, and how to identify good test code. Come away with practical fundamentals and coding guidelines to keep your own test automation project from sinking beneath the murky waters.

1. BT9
Test Automation
11/17/2016 1:30:00 PM
Conquer the Murky Waters of Test
Automation
Presented by:
Brian Saylor
Scripps Networks Interactive
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888--‐268--‐8770 ·∙ 904--‐278--‐0524 - info@techwell.com - http://www.stareast.techwell.com/

2. Brian Saylor
Scripps Networks Interactive
Brian Saylor is currently the automation architect for Scripps Networks
Interactive where he oversees plans and strategies for test, release, environment,
and data integrity automation for the Scripps Networks family of websites—
HGTV.com, FoodNetwork.com, and TravelChannel.com, and others. During his
software development career Brian has held many roles including senior software
engineer, software development manager, and subject matter expert for web
technologies and content delivery networks in both large companies and small
startups. He is recognized for his ability to solve complex technical challenges in a
practical manner.

3. CONQUER THE MURKY WATERS
OF TEST AUTOMATION
Brian Saylor
@ChaosComputing

4. INTRODUCTION
OVERVIEW
▸ Who am I?
▸ Why am I here?
▸ Why are you here?

6. OBJECTIVES
WHY DO WE AUTOMATE?
1. Increase overall product quality
2. Increase development throughput
3. Decrease turnaround time for testing
4. Focus more attention on development of features
5. Decrease cost
6. Provide test coverage for features that are difficult or time-
consuming to test
7. Reduce fear of change
8. My boss told me too

7. OBJECTIVES
WHY DO WE AUTOMATE?
1. Increase overall product quality
2. Increase development throughput
3. Decrease turnaround time for testing
4. Focus more attention on development of features
5. Decrease cost
6. Provide test coverage for features that are difficult or time-
consuming to test
7. Reduce fear of change
8. My boss told me too

8. OBJECTIVES
WHY DO WE AUTOMATE?
1. Increase overall product quality
2. Increase development throughput
3. Decrease turnaround time for testing
4. Focus more attention on development of features
5. Decrease cost
6. Provide test coverage for features that are difficult or time-
consuming to test
7. Reduce fear of change
8. My boss told me too

9. OBJECTIVES
WHY DO WE AUTOMATE?
1. Increase overall product quality
2. Increase development throughput
3. Decrease turnaround time for testing
4. Focus more attention on development of features
5. Decrease cost
6. Provide test coverage for features that are difficult or time-
consuming to test
7. Reduce fear of change
8. My boss told me too

10. OBJECTIVES
CATEGORIES
▸ Increase throughput / Decrease turnaround time
▸ Increase product quality
▸ Other

11. GOALS
TEST AUTOMATION GOALS
▸ Reliability
▸ Performance
▸ Coverage
▸ Prioritization
▸ Workflow
▸ Cost efficiency

12. GOALS

13. RELIABILITY

14. THE EXTENT TO WHICH AN EXPERIMENT,
TEST, OR MEASURING PROCEDURE
YIELDS THE SAME RESULTS ON REPEATED
TRIALS
Merriam Webster
RELIABILITY

15. ACTIONS
RELIABILITY
▸ Separate data/code/model
▸ Remove variables
▸ Remove third-party dependencies
▸ Consolidate duplicate or common code
▸ Controlled test data
▸ Test!
TEST
YOUR
SHIT!

16. PERFORMANT

17. THE ABILITY TO DO SOMETHING OR
PRODUCE SOMETHING WITHOUT
WASTING MATERIALS, TIME, OR ENERGY
Merriam Webster
EFFICIENT

18. ACTIONS
PERFORMANT
▸ Fail Fast
▸ Parallel execution
▸ Streamline tests
▸ Eliminate redundant tests
▸ Eliminate low value tests

19. PRIORITISE

20. ACTIONS
PRIORITIZATION
▸ Eliminate tests that should not be automated
▸ Development effort
▸ Business value

21. PRIORITIZATION
BUSINESS VALUE
▸ Impacted Users
▸ Financial Impact
▸ Failure Probability
▸ Incremental Impact

22. ACTIONS
COVERAGE

23. WORKFLOW

24. COST EFFICIENCY

25. TEXT
CODE REVIEW CRITERIA
▸ Validate coverage
▸ Waits
▸ Hard coded values
▸ Third party dependencies
▸ Separation of test data /
code / model
▸ Performance
▸ Assertion Messages
▸ Complex Code
▸ Race Conditions
▸ Generic code

26. SUMMARY
OBJECTIVES
THROUGHPUT
TURN-AROUND
TIME
QUALITY
COST

27. ON TO THE MAGIC LAND OF DEV OPS
▸ Brian Saylor
▸ @ChaosComputing
▸ ChaosInComputing.com
▸ Resources
▸ Calculating Business Value
▸ http://goo.gl/5us6DA
▸ Peer Review Criteria
▸ http://goo.gl/JnmitP