DevOps promotes continuous integration, continuous testing, and continuous deployment. And anything that breaks this continuity is a potential bottleneck. In many organizations, testing becomes that bottleneck for one or all of the following reasons: unstable test environment, lack of good test data, and manual test processes. Tanya Kravtsov explores why treating test automation as a separate entity from development often results in a DevOps failure. She explores how integrating test automation with product development—including data, environment, and results analysis—will help you succeed. According to the Theory of Constraints, a chain is no stronger than its weakest link. Any manual process in software testing that disrupts the continuous build, test, deploy cycle will either extend the time to market or add to technical debt due to insufficient testing. Learn to identify the bottlenecks in testing, and explore tools and techniques available to address these bottlenecks. Come away with a vision of what your ideal “future state” should look like and ideas of how to achieve the optimal DevOps state for your organization.

1.
W17
DevOps
&
Testing
5/4/16
15:00
The
Road
to
DevOps:
Data,
Environment,
and
Test
Automation
Presented
by:
Tanya
Kravtsov
Audible
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.stareast.techwell.com/

2.
Tanya
Kravtsov
Audible
Tanya
Kravtsov
has
recently
joined
Audible
as
a
Director
of
QA.Â
Previously,
as
the
head
of
automation
and
continuous
delivery
at
ROKITT,Â
TanyaÂ
focused
on
internal
Automation
and
DevOps
practices
and
worked
with
customers
to
transform
their
development
and
testing
processes.
Tanya
is
passionate
about
process
automation,
which
encompasses
test,
data,
and
environment
automation,
as
well
as
continuous
integration
and
continuous
delivery.
She
is
a
founder
of
theÂ
DevOpsQA
NJÂ
Meetup
group
and
a
speaker
at
events
including
QUEST,
TestBash,
and
the
Jenkins
User
Conference.
Follow
her
on
TwitterÂ
@DevOpsQA.

3. The Road to DevOps:
Data, Environment, andTestAutomation
Star East 2016

4. Agenda
 Introduction
 Definition of Continuous Delivery
 10 Steps
 Target State

5. Tanya Kravtsov
• Director of QA at
• Head of Automation and Continuous Delivery at
• Founder of DevOpsQA NJ Meetup group
• Developer turned Tester
@DevOpsQA

6. Definition of DevOps
ULTURE
UTOMATION
ONITORING
HARING

7. Star East 2016

8. “Any improvements made anywhere besides the bottleneck are an illusion”
Identify Bottlenecks - Why does it matter?

9. Bottleneck IdentificationTechniques
 Play the Speed Boat Game
 Identify Anchors holding back your process
 Draw theWorkflow
 Identify all players in the systems
 Identify all handovers between the players
 Communicate
 Brainstorm
 Mind Maps
 Interviews
 Retrospectives

10. Star East 2016

11.  Automated Environment Setup process
 Infrastructure as Code
 Automated Configuration Management
 Virtualization
 Infrastructure Monitoring
 Availability Monitoring
 CPU Utilization
Test Environment Setup and Monitoring

12. Automated SmokeTest
 Automated Environment Readiness test
 Application Server
 Web Server
 Database Server
 Automation Framework Artifacts
 Authentication
 Virtual Machines
 Automated Product Readiness test
 Validate critical functionality

13. Star East 2016

14.  AutomatedTest Data Management
 Data Discovery
 SyntheticTest Data
 Data Subsetting
 Data Masking
 DataCleanup
 Integrated Automation andTest Data solution
 UtilizeTDM tool APIs to retrieve data at runtime
 Generate fresh data before each execution
 Automatically clean up data at the end of the run
Test Data Management

15. Star East 2016

16. Automated UnitTesting
 UnitTests
 Short independent tests
 Tests only fail when a bug is introduced
 When tests fail, we know the reason
 TDD
1. WriteTest First
2. Write enough code to make the first test fail
3. Fix the code to make the test pass
4. Repeat

17. Automated Build
 Automate
 Compiling computer code into binary
 Packaging binary code
 Deploying to test and production environments
 Creating documentation
 Setup CI Server
1. Install Jenkins CI
2. Configure Jenkins Server
3. Install Jenkins Plugins
4. ConfigureVersionControl System
5. Setup Jenkins Job

18. Automated Code Quality Metrics
 Static Analysis
 Security vulnerabilities
 Common and critical defects
 Coding Standards
 Memory Leaks
 Code Coverage Analysis
 Measure UnitTest Coverage
 Measure application-level test coverage
 Code Reviews
 Pull Requests
 Pair Programming

19. Automated Regression
 Test Automation
 AutomatedWeb ServicesTesting
 Automated UITesting
 Parallel Execution
 VirtualizedTest Lab
 Central Management Dashboard
 AutomatedTest Data Management
 Retrieval
 Generation
 Cleanup
 SelectiveTesting
 Pairwise testing
 SearchableTest Case Repository

20. Star East 2016

21. Automated Analysis and Reporting
 AutomatedTest Results Analysis
 Consolidated Results
 FailureAnalysis
 Defect Logging
 Automated Results Reporting
 Automated Email Notifications
 Reporting Dashboard
 Excel, PowerPoint,ALM
 RootCause Analysis

22. QUEST 2016

23. Automation and DevOps Myths
 Testing can be 100% Automated
 Automation is a lot faster than manual
testing
 Automation should be maintenance free
 Automation can be created without any
scripting
 Adapting DevOps will result in 50 builds a
day fully tested and ready for production
21

24. Target State
Communication