Advanced Continuous Test Automation presentation given by Marc Hornbeek, Sr. Solutions Architect for Spirent at the IEEE meeting Buenaventura chapter March 9, 2016.

1. Advanced Continuous Test
Automation
Marc Hornbeek
Sr. Solutions Architect

2. 2Spirent Communications
Agenda - Advanced Continuous Test Automation
• Evolution of automated testing
• Requirements of DevOps CT at scale
• Implementation approaches
• Summary

3. 3Spirent Communications
Sept 9th, 2015
IEEE Buenaventura Section CS chapter meeting

4. 4Spirent Communications
Etymology
 “Test” (verb) – early 17th century
 “Automatic” – mid-18th century
Early fully automated systems in “production”
 18th century clock production
Automated Testing History – Beginnings

5. 5Spirent Communications
Automated Testing History – Mechanical mid-1700’s
1736: Sea trial of ‘Harrison 1 “ marine chronometer solving the problem
of calculating longitude while at sea

6. 6Spirent Communications
Automated Telephone
Exchange – 1891
Automated Test History - Electric-Mechanical – late 1800’s

7. 7Spirent Communications
1943
“Colossus”
Digital
Computer
Code
Breaker
1939
A. Turing’s
“Bombe”
Enigma
Code
Breaker
1946
“ENIAC”
Ballistics
Tables
and H-
Bomb
Automated Testing History - Programming mid -1900’s
1954
Mass-
produced
computer IBM
650
Debugging and testing were not considered separate concepts

8. 8Spirent Communications
• 1961 - Computer Programming Fundamentals, Leeds, Weinberg
• 1963 - Flowchart symbols standardized, Rossheim
• 1964 - First Basic program, Dartmouth College
• 1965 – IBM 360 – 1 MLOC
• 1967 - Functional Testing of Control Programs, IBM
• 1967 – “Software Engineering”, NATO
• 1968 – “Software Quality Assurance”, NATO
1960’s – Large scale software systems, SQA emerges
Scripting
C
JCL
Unix Shell
Apollo 11
145
KLOC

9. 9Spirent Communications
• 1971 - IEEE Computer Society founded
• 1972 - C language, Dennis Ritchie and Brian Kernighan
• 1974 - MIL-S-52779 Software Quality Program Requirements
• 1975 – Microsoft founded
• 1976 – Apple founded
• 1976 - Software reliability : principles and practices, Glenford Myers
• 1976 - Design and Code Inspections, Michael Fagan
• 1976 - Cyclomatic complexity metric, Tom McCabe
• 1979 - The Art of Software Testing, Glenford Myers 1979
• 1983 - IEEE 829 Standard for Software Test
1970’s – Software testing science foundations

10. 10Spirent Communications
A Network Test Environment for Packet Switching
Networks, IEEC&E, 1983
1980’s/90’s – Software and Protocol test tools
• Code coverage tools
• Static analysis tools
• Unit test tools
• Load test tools
• UI test tools
• Data protocol testers
• Advanced scripting
LINT
QTP

11. 11Spirent Communications
2004+ Test infrastructure automation
11
• Test topology automation
• User and resource scheduling
• Lab resource management
VelocityLab Manager

12. 2010+ Lab-as-a-Service (LaaS)
• Elastic on-demand
resources
• Test infrastructure
as code
• Virtualized tools
Private or public or hybrid

13. 13Spirent Communications
Waterfall Test Process - 1956
Most tests are
concentrated near
the end of the
pipeline
Bugs have a long
find-to-fix duration

14. 14Spirent Communications
Agile Test Process : 1995
Collaborative teams test each iteration of a product development in “Sprints”.
Agile emphasizes test automation. Does not prescribe infrastructures for
integrating test activities across the development-to-delivery infrastructure.
Reference: “Agile Testing: A Practical Guide for Testers and Agile Teams”
Extreme programming & Test Driven Development (TDD)

15. 15Spirent Communications
DevOps Continuous Testing - 2007
Design Right Shift Left
Development
Wants agility
Operations
Wants stability
Continuously eliminate
process friction and waste
13.4 million google hits (was 4M 9/2015)
CI CD
CM
CT
“Infrastructure as code”

16. 16Spirent Communications
Agenda - Advanced Continuous Test Automation
• Evolution of automated testing
• Requirements of DevOps CT at scale
• Implementation approaches
• Summary

17. 17Spirent Communications
Continuous testing is not a phase in the DevOps pipeline
CT is implemented as part of all DevOps pipeline stages in accordance
with a continuous testing strategy specific to a product and
organization.
Myth Buster: CT is NOT a Phase !

18. 18Spirent Communications
Test Automation
 Versions
 Dependencies
 Tests
 Creation methods
 Type of Test
 Pipeline stages
Lab Automation
 Resources
 Stakeholders
 Administration
 Topologies
 Production variations
Dimensions of
Test and Lab
automation

19. 19Spirent Communications
Continuous Testing Types
• Unit tests: white box test that validates code logic
• Static analysis tests: check source code logic and consistency
• Dynamic analysis tests: test the code performance
• Code coverage tests: measure % of code tested
• Integration tests: test that code changes work after a code merge
• Compatibility tests: Test feature interactions
• Conformance tests: Test conformance to standards
• Usability tests: Test user experience
• Application tests: Test the application functions
• API tests: Test the Application Programming Interface
• Regression tests: Verify changes didn’t break something
• System test: Test system wide-features
• Performance tests: Test capacity, throughput and delay attributes
• Security tests: Test that security features operate
• Administration tests: Test administration features

20. 20Spirent Communications
Continuous Test Pipeline Stages
Pre-Flight Continuous Integration (CI) Continuous Delivery (CD)

21. 21Spirent Communications
Large Scale DevOps CT Test Environment
• Many short cycles
• Aggregated changes
• Large pool of tests
• Orchestrated pipeline
• Elastic infrastructure
applies at every phase and in every process

22. 22Spirent Communications
Sizing Test and Lab Automation

23. 23Spirent Communications
Sizing Test and Lab Automation

24. 24Spirent Communications
Agenda - Advanced Continuous Test Automation
• Evolution of automated testing
• Requirements of DevOps CT at scale
• Implementation approaches
• Summary

25. 25Spirent Communications
Design for Automated Continuous Testing
 Test Driven Development practices
 Layer by layer isolation:
 initiate a class, independent classes, verification
 Feature toggles
 GUI registration
 RESTful APIs
 Integrated agents
 Tool plugins
Kent Beck said, "Code that isn't tested doesn't work -
this seems to be the safe assumption."
However, not all applications are easy to test.

26. 26Spirent Communications
Five Tenets of Continuous Testing
Test fast
• Design for test
• Automate at start
• Production environment
• Automated tests
• Incremental
• Infrastructure as code
• Elastic scaling
• Smart scheduling
• Rugged lab
• Analytics
• Implications for test and lab automation
CT
Tenets

27. 27Spirent Communications
Best Practices
• Continuous Testing
• Test Automation
• Test scripting
• Lab automation

28. 28Spirent Communications
CT Best Practices

29. 29Spirent Communications
Test Automation Best Practices

30. 30Spirent Communications

31. 31Spirent Communications
Lab-as-a-Service Best Practices

32. 32Spirent Communications
Advanced Continuous Testing
Automated
test
selection
and analysis

33. 33Spirent Communications
http://www.slideshare.net/Spirent/automated-continuous-test-selection-
methods-for-large-scale-devops

34. 34Spirent Communications
Advanced CT Test Selection Methods Focus
The test selection methods operate at different points
of the code-build-test pipeline
Building TestingBuild Result
Coverage
CodeCoding

35. 35Spirent Communications
Test Trends Test Selection Method
Cumulative code-to-test verdict correlation
factors (CF) are saved in a table for all code
modules and tests.
CF numbers are adjusted when tests are run
for a code module. (Note: Values are neutral
for new tests or code set to neutral value.)
Test cases are selected which have the highest
correlation to the changed code modules.
Experiences of Automated Testing in Continuous Change Management - MERGE 2014
conference
http://www.slideshare.net/perforce/automated-testing-in-continuous-change-management

36. Automated Continuous Testing Optimization - Simulation
Optimum
process
• High backlog rate
• Early stages have higher fail rate
• Process time is equal = continuous flow
What pipeline of processes yields the highest backlog depletion rate,
highest number of fixes, and highest number of changes per time unit?
What to control?
• Process time
• Pass rate
• Fail rate

37. 37Spirent Communications
Consequences of not doing CT properly
 Culture conflicts
 False negative and false positive
test results
 Test escapes
2015 software glitches
3,200 US prisoners released early
HSBC 275,000 payments failed
114,720 wrong divorce payments
London trading down for two hours
600K RBS paychecks
Nissan 1 million recall
60% of Starbucks stores closed
F-35 Fighter target error

38. Test and Lab Automation
Micro-Services Architecture
Test and Lab Automation Tools
Framework

39. 39Spirent Communications
CT-Ready Tools
Continuous
Deployment
(CD)
Continuous
Integration
(CI)
Plug-ins
Plug-ins
SUTTools
Lab Management
Physical, Virtual, Hybrid Lab
Analytics ALM
Orchestration
Continuous Test (CT)
CCM
Select tools framework to match your goals for high
availability and virtual environments
Select tools with RESTful APIs, plugins
Warning !

40. 40Spirent Communications
Success Story
Metric
Major release
(#months)
Minor release
(#weeks)
# Features
Defects
Integrations / day
Tests / day
Automated tests
After
3
2
150
10
100
10
85%
Before
6
4
113
1260
0.5
0.3
5%

41. 41Spirent Communications
Agenda - Advanced Continuous Test Automation
• Evolution of automated testing
• Requirements of DevOps CT at scale
• Implementation approaches
• Summary

42. 42Spirent Communications
The future of Automated Continuous Testing?
Self optimizing processes
• Design for test
• Self-test
• Live Test
• Analytics
• Predictive
• Automated remediation

43. 43Spirent Communications
www.devops.com Hornbeek blogs
Continuous Automated Testing
References
www.spirent.com/solutions

44. 44Spirent Communications
Summary
 Advanced Continuous Test
Automation is required for
DevOps at scale
 Make the effort to understand and
use best practices
 Preferred implementations use
Lab- as-a-Service
 The future is end-to-end process
self-optimization

45. 45Spirent Communications
© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name
“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.
spirent.com
Thank you
Marc Hornbeek
Senior Solutions Architect
Marc.Hornbeek@spirent.com
Twitter: mhexcalibur
https://www.linkedin.com/in/marchornbeek