As IT organizations adopt a DevOps strategy, continuous testing (CT) becomes a key ingredient of the DevOps ecosystem. CT enables faster release cycles, more changes per release, upfront isolation of risks, and reduced operations costs. The approach to scale the traditional automation testing infrastructure, test environments, and test data management requires a culture shift using new tools and techniques. Sujay Honnamane discusses a CT strategy for aspiring and already implemented DevOps organizations. Sujay shares examples of tools, techniques, and practical solutions that include continuous integration using the Jenkins CI server, service virtualization through CA Lisa tools, automated code coverage analysis to create impact-based tests, automated test script load balancing for effective use of test environments, and faster test cycles, providing a holistic approach/workflow for CT. Sujay and his teams have successfully implemented CT for several clients in their DevOps journey to achieve a repeatable and highly predictable software delivery process.

1. Continuous Testing: A Key to DevOps Success
Sujay Honnamane
Test Director
Cognizant Technology Solutions
1
© 2015 Cognizant

2. 2
Continuous Delivery is critical for an enterprise
Dev & Test
Teams
Operation
TeamsGap
Evolving
Customer
Expectations
Faster Time to Market
Need for higher quality
Changing Business
Requirements
Business Agility, performance,
Availability and Stability
Leads to Continuous Delivery Challenges…
Lack of collaboration
Infrequent, manual, error prone
deployments
Long QA feedback Loop
Mobile Cloud Social Big Data
Connected
Systems
Emerging Technologies
DevOps
Deploy
Test Continuous Delivery with DevOps

3. Let’s face it:
• Applications that companies deal with
actually contain very complicated business
logic implementations than what Facebooks,
Twitters and Netflix/es are
• Years of legacy – these applications are out
there for more than decade!
• There are patterns adopted by Development,
Operations and QA and organization’s
expectations revolve around it!
• Agile revolutionized development to the
extent – Operations remained in silo and QA
remained bottleneck
Image source : https://paimeiitguy.wordpress.com/2008/07/28/a-beacon-of-hope-for-the-keep-it-simple-stupid-way-of-working/
Reality Check – Simplicity is highly overrated by Unicorns!
3

4. • Traditional Ops – putting out fires, deal with recurring failures & urgent fixes
• Test Environment overhead– Each environment is unique by zillion different ways
• Several thousands of interdependent regression cases running through UI layer for
several days
• Longer test cycles & manual handoffs – much slower feedback
• Loosely defined test data management strategy
• Downstream/Upstream dependencies
• Silos within silos inhibiting cross functional leverage
Image source: http://legacyportfolio.co.uk/blog/2012/06/08/what-are-the-challenges-facing-corporate-
occupiers-today/
Challenges for horses today
4

5. Spot the opportunities & Correct them – One at a time
5

6. OPS
What in the
world?
Dev
QAProduct owner
• Build failure rate > 60%
• OPS : DEV problem , DEV: OPS problem
• Avg. time for build deployment > 2 days
Our teams..Chaos – First DevOps Meeting
6

7. There is a better way!
And QA can help!!
Oh Really?
Who cares?
Deployment is
painful task, these
folks won’t ever
get it!
QA Architect
QA’s first attempt to break the silos
OPS
Dev
Product owner
7

8. Someone
is making
sense!!
No kidding! We work
hard to make
everyone’s life easy
Product owner
Hmm! We
are
listening!
A start..constructive discussion – QA gets Dev in alignment
OPS
Dev
8

9. • QA has greater insights to both
Development and Operations side of
the worlds
• Deployment delays hurts QA more
than any one
• Continuous Integration is first step in
right direction – QA and Dev
collaboration can make it happen
Can QA be the DevOps enabler? Yes, we can !
QA
9

10. DevQA
With group of motivated individuals from both QA and Dev
Goal Statement:
“Predictable builds, deployments and quality
assurance procedure that delivers the working
software from developer’s IDE to the hands of
business users with click of button.”
And, DevQA journey began!
Image source: http://encontrosdesign.info/how-team-building-exercises-can-help-your-
business/
10

11. WE, the people:
Collaborative –Working together to build innovative solutions
Open minded - Experiment,fail,learn and Adapt
Tool agnostic – solution driven than driven by tool
Fearless – Makes problem visible
OneTeam spirit – Combined ownership of problems, no blame
Love coding – ruby, python, ant whatever comes our way
DevQA team’s attitude
Image source : http://voices.whitman.syr.edu/student-life/undergraduate/teamwork-makes-the-
dream-work
11

12. SVN Server
Automation Team
Jenkins Server
Slave Agents
Dev / QA
Environments
Stakeholders
Continuous Integration – Dev / QA Environments
Updated
Automation
Script
Triggers Build Verification
Tests
 Health Checks
 Smoke Tests
 Intelligent regression
Results
Code Deployments in
Dev / QA environments
Accessing Environments
Build verification Results to
stakeholders through email
Dev code:
New User
Stories /
Defect Fixes –
Code Commit
Polling the
Updated Code
Updated
Automation
Script
Dev Team
Automated Build Creation
Automated Security Scan
Unit Test
Smoke Results
Critical Scenario Results
Regression Results
Static Code analysis
Build Quality
In
Code Coverage analysis
Instant
feedback on
quality
 Virtualized
services
 Production
data subset
12

13. © 2014 Cognizant
Continuous Integration - Critical Success Factors
Integrate,Build,DeployandValidate
1 2 3 Continuous Testing
4 ServiceVirtualization 5 Environments on Demand
“Shift Left” QA and Dev synchronization
13
13

14. Existing full regression running across three different tools took us 50hrs to complete, we needed to build something intelligent that validates
relevant stuff for in shortest possible time! Rest of the regression runs over the weekend!
It addresses most common challengesQA organization face today:
 Legacy automation built on commercial tools over the period of time runs
slow
 Often, automation is built over altogether in different tools to cover for UI,
APIs, Mobile coverage etc.
 Business influence over regression definition often push testing team to
build long running tests
Intelligent Regression Model – The key success factor
14

15. Intelligent regression model – Key considerations
Image source: http://www.signaturelandscaping.ca/
• Dynamic regression scope for each build based on type of
code change - Automatically generated release notes
feeds in to regression job
• Test cases are traceable to the code files
• Intelligent framework capable of turning test scenario
ON/OFF based on application change
• Risk based approach with close to zero risk
• Centralized repository of code-to- test case traceability
15

16. DevOps Transformation – QA Levers
AutomatedTest design through intelligent model based
framework
Faster feedback through
• Well defined heartbeat tests & deployment verification test
• Massive parallelization of tests powered by efficient load
balancing of tests between nodes
• Shift from UI based tests to Non-UI based tests
• Intelligent test suite
Reduced testing dependency
• Service virtualization
• Adoption to more isolated tests over End to end tests
• Test Data Management
16

17. Conclusion:This is Dev Ops !
17

18. THANK YOU
18