Continuous Testing in DevOps

DT2
DevOps & Testing
6/8/2017 10:00:00 AM
DT2 Continuous Testing in DevOps
Presented by:
Michael Hackett
LogiGear Corp
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - https://www.techwell.com/

Michael Hackett
LogiGear Corp
The head of the training, consulting, and publishing division of LogiGear
Corporation, Michael Hackett has twenty-five years of experience with software
engineering and testing in banking, securities, healthcare, and consumer
electronics. He has trained teams in sixteen countries and runs a global
consulting practice for organizations wanting to increase quality and productivity.
Michael authored Testing Applications on the Web and Global Software Test
Automation. A Certified ScrumMaster, Michael is an instructor at UC Santa Cruz
and Berkeley Extension, and four universities outside the US.

5/25/2017
1
Continuous Testing in DevOps
Michael Hackett
Senior VP at LogiGear Corporation
Agile Dev West
Las Vegas
June 2017
©2017 LogiGear Corporation 1 ©2017 LogiGear Corporation 2

5/25/2017
2
1. In this presentation… What is DevOps?
– All the Continuous: Continuous Testing, Continuous Delivery
2. Production Line, Deployment Pipeline Mentality
3. New Testing Strategy for Continuous Testing
– How does Testing fit in?
– What to Fix
– Test Automation Suites
4. JumpStart and Summary
Seminar Overview

5/25/2017
3
Business ERP software.
B2B Global, Distributed teams.
3 years ago: “Let’s do DevOps!”
• Dev and Ops did not particularly like each other. No respect for each
other’s work.
• Did not want to be at daily stand up.
• No culture change-just add tools “Start today- just do it. “
• Dev did not want to fix Ops problems.
• Ops resented getting forced into the ambiguous “mess” of development.
• Dev wanted to wash their hands of the code once they hit their Done
criteria.
• Attempt at DevOps was a total failure.
• They stopped doing it. Exec demoted- embarrassing for all dev and ops
teams.
New completely different attempt began this year.
• Very slow.
• “It’s a Journey.”
• Aiming to change culture.
Case Study #1 (part 1)
©2017 LogiGear Corporation 5
Get a full understanding of where Testing is inserted into
the DevOps/Continuous Delivery Pipeline.
Don’t let testing be a bottleneck—Move your testing out
of a sprint mentality, into a deployment mentality.
Understand the cultural and practice shifts necessary for
the move.
Get Moving! Jumpstart Continuous Testing in
Continuous Delivery
Goals of this Session

5/25/2017
4
What is DevOps?
DevOps
What is an DevOps? This is too easy!

5/25/2017
5
What is DevOps? Reality.
“Agile for Ops”
“Agile on Steroids”
Lean, Production Line Product Development
Agile is Process and Product
DevOps is Process, Product and
Infrastructure
What is DevOps?

5/25/2017
6
Just like ScrumButts and Agile Falls…
We already have
DevOops
and
DevFlops
What is DevOps?
DevOps is a journey.
It means many things to different people.
All these descriptions of DevOps are important. They will
clue us in on what continuous testing is about and what the
team expects of us.
DevOps Overview

5/25/2017
7
Many of us are already on this journey.
For us today, focused on Continuous Testing,
DevOps is:
Quality at every Step
Shift Left
More and more Task and Test Automation
Team Culture Change
Much smaller releases
Tool chain revolutionizing practices
DevOps Overview
Business Needs
• Production Line, development is not an art form
• Predictability and Scheduling
• Constant Feedback
Combining XP and Lean Principles most impacting
Testing in Continuous Delivery:
• Small Releases
• Continuous Integration
• Collective Ownership
• Quality at every Step
• Deliver as fast as Possible
• See the Whole
What are some DevOps Culture changes?

5/25/2017
8
What are some DevOps Culture changes?
People, Process, Product all collaborate and work together.
Operations is now part of the application development
Tools eliminate or reduce environment, data, installation,
provisioning and deployment challenges.
Move to cloud technologies
Everyone is responsible for deployment. Deployment is no
longer only Ops job.
Product is always at Release Quality. Eliminate technical
debt.
15
Jez Humble and David Farley
Continuous Delivery: Reliable Software
Releases through Build, Test, and
Deployment Automation
Read:

5/25/2017
9
Pre-Reqs:
• Some understanding of DevOps
• Understanding of Testing
• Lean
• XP
• Tool Chains
DevOps Overview
“Stop saying DevOps”
• DevOps has evolved to CD.
• CI focuses on Dev and Test Teams, CD
focuses on Ops/Deployment issues.
• Where CI ends, CD picks up.
• CD is possible only after Continuous
Integration, then Continuous Testing.
• We will look at insertion points for testing
Testing in Continuous Delivery

5/25/2017
10
Continuous Integration is a core XP
(extreme Programming) practice of building
often. Very often. To be successful using
Scrum/Agile, this process needs to be
automated.
Most commonly:
• Build on commit (latest and greatest).
• Build daily.
• Build on demand.
CI Overview
Level of CI process
1. Auto build
2. Rerun unit tests
3. Rerun API and UI driven smoke tests
4. Make some VMs
5. Rerun smoke tests on VMs
6. Run full regression once or on all VMs.
Send a report after final phase
CI Overview

5/25/2017
11
The Big Picture
21
Continuous delivery - Continuous delivery is a series of practices
designed to ensure that code can be rapidly and safely deployed to
production by delivering every change to a production-like
environment and ensuring business applications and services
function as expected through rigorous automated testing. Since
every change is delivered to a staging environment using complete
automation, you can have confidence the application can be
deployed to production with a push of a button when the business is
ready.
Continuous deployment - Continuous deployment is the next step of
continuous delivery: Every change that passes the automated tests
is deployed to production automatically. Continuous deployment
should be the goal of most companies that are not constrained by
regulatory or other requirements.
Continuous Everything!

5/25/2017
12
Continuous Testing:
• All the testing Test Teams know
• Perhaps testing in production
• Running other people’s tests/other people
running our tests.
• And it’s all automated.
• But its not merely re-running your “full
regression” over and over and over.
Continuous Everything!

5/25/2017
13
Continuous Delivery Pipeline
DevOps
Very Large Financial Service
Global, Not co-located teams.
“We are now going to do DevOps.”
• Agile implementation took so long and misunderstood, many
teams gave up. Some teams still use traditional (old) style
development.
• One foot in waterfall, one foot in agile. Pure AgileFalls.
– But agile practices are still a mess.
• They have no clue what devops is really about.
• Teams are afraid devops means more pressure to release faster
with no training or support.
• Very little automation. What little they have is not effective.
• Begin by group discussions/trainings on what devops is.
• “It’s a Journey.” Maybe a long journey. Incremental change.
• Aiming to change culture and support.
Case Study #2

5/25/2017
14
Flow
• Business has greater control of the deployment process
• Tools are enabling significant advances in SDLC
• Higher quality because of focus on testing
Production Line: Deployment Pipeline
CD
• Every task/process is automated from source control to
delivery
• Its an automate production line. It provides for a constant flow
of value (new functionality, change/fixes) to production.
• It extends the build branch idea into a full process flow
through every development task. Tasks fully automated.
• Providing immediate feedback at each step.
What is CD?

5/25/2017
15
All this testing is to keep the pipeline moving along.
• Image from XebiaLabs
• https://xebia.github.io/cd-with-docker/img/continuous-deployment-pipeline.png
CD Pipeline
• But not “our” big concern
• Our Automation has to fit in, easily, run by anyone
The Tool Chain is important

5/25/2017
16
CD is not only about tools!
It seems so these days.
Lean Practice of Quality at Every Step
• A poka-yoke is any mechanism in a lean manufacturing
process that helps an equipment operator avoid (yokeru)
mistakes (poka). Its purpose is to eliminate product defects
by preventing, correcting, or drawing attention to human
errors as they occur. (Wikipedia.org)
What is CD?
What is Testing’s role in this pipeline?
• At each step along the pipeline flow, some type of validation
or verification needs to happen.
• Automate as much as you can.
• Keep the pipeline moving! No bottlenecks from testing.
• Testing gives immediate feedback.
– A measure of how good testing is in CD- how immediate is
feedback?
What is CD?

5/25/2017
17
All this testing is to keep the pipeline moving along.
• http://www.informit.com/articles/article.aspx?p=1621865&seqNum=2
Continuous Delivery: Anatomy of the Deployment Pipeline
By Jez Humble and David Farley
The Tool and Ops Revolution
Powering the Pipeline
Quality at Every Step
The Lean Principle of “Quality at Every Step”
Its the basis of DevOps as Continuous Testing

5/25/2017
18
A few words about Environments:
Out of Scope today…
• Infrastructure-as-Code
• VMs- VMware, Hyper-V, VMM
• Docker or other- Containers
• Cloud- AWS, Azure
• IaaS, EaaS…
Solve Data and Environment Issues
Retail-Consumer Product
US, Co-located teams
New Engineering Manager wants to begin Continuous Testing.
• Test teams have to plead with Dev or IT/Ops or anyone to update their
servers.
– Finger pointing between Dev and IT to manage test servers.
– No super skilled system-type testers on team.
• Highly integrated system- legacy and newer systems
– But not more than many companies
• Test teams “finds” bugs that are not really there.
• Difficulty reproducing errors.
• Miss bugs. Lots and lots of support calls (their main problem)
• The data is a mess: old, rarely refreshed.
Journey begins
• IT will “experiment with VMs” on test team first.
• Get more tech skill on test team.
• Change culture- other teams forced to see how server and data
problems for test team were their problems to fix.
Case Study #3

5/25/2017
19
Virtualization, cloud
• If you are not fluent and comfortable using Virtual Machines/
VMs- learn about it! Practice/Use your organization’s VM tool.
Virtualization will be an important key to Continuous Testing
success.
• Get tech help
• Have IT do all the environments for you. Collaborate.
• Some teams run everything on VMs.
• Cloud- a goal of many DevOps efforts is to fully leverage cloud
and IaaS/EaaS/PaaS
– Understand the cloud impact on your tools, reporting, tests…
• Many organizations use VMs and Cloud to solve the environment
and data problems listed above.
Environment and Data problems?
Environments and Data-
• If your team has problems setting up environments, getting
consistent environments – fix this right away.
• If your team has data problems- refreshes, updates, mirroring,
loading, etc. – fix this right away.
• Env and Data cause problems for many test teams. These
problems will prevent CI/CT success. You need to overcome data
and environment problems.
• Get tech help if you do not have it on your team.
• Better test environment and test data management.
Environment and Data problems?

5/25/2017
20
The New Test Strategy
DevOps Overview

5/25/2017
21
So far:
• Quality at Every Step
• Shift Left
• Even more distributed testing/ownership of quality
• Extend CI
• Regression all automated and fast
The New Test Strategy
Financial Service
Consumer Product
US, Co-located teams.
“We are going to do DevOps.”
• Ops happens at the end- Dev/Test have no idea:
– Ops has too much to do manually. Provisioning, user lists, databases. 3rd party data
loaded. Full install.
– All nite- it takes a very long time. days and days. The business need does not matter.
… Ops takes a big long manual process.
– 2am 3am 4am phone calls.
– Dev and Test have no clue what other people were doing. Many groups still in silos.
– Some testing tasks are put off til “Ops or IT” can get to them.
• CI was in a few places but many teams not heard of it. No one else could
run the CI past autobuild.
• Of course, Ops finds load, performance, login, security problems.
– “Why didn’t you find these!” Throws things back to dev. Finger pointing. Testing sits on
the sidelines. Finger pointing. Gets delayed days, POs scream, point fingers.
• Of course, a lot of quality problems.

5/25/2017
22
What has to be perfect?
• Same sprint automation
• Definition of Done
• Collaboration
• Estimating
• Testing skills
• Feedback
• Interaction with the business/PO
• Great User Stories
• Unit testing
…..no Scrumbutts!
Agile has to be perfect- already
Continuous Testing is in addition to and different from testing
individual user stories or testing in sprints. This testing will
continue as a mix of manual then automated.
Continuous Testing

5/25/2017
23
Re-examine your test strategy:
• What are you testing and where?
• What is best to test at various interfaces?
• What is better automated vs. manual?
• What do you run different servers?
Continuous Testing
What
Environment
Dev Testing
Staging or
Pre-production
Live or
Production
Who Developers Test team Test & Ops Test & Ops
Purpose
Introduce new
code, test in
isolation, test
integrated code,
small data, not fully
functional
Test through UI, full system,
integration on limited
environment, bug finding,
experiment, discovery of
how system works,
Production-like, Fully
enabled, every piece
of the system in
pace, policies, bigger
or full data, no
“traffic”
Monitor live
system
Tasks
Unit testing,
component,
integration, API
UI, End2End, workflow,
regression, bug verification,
system testing, small,
limited data, API,
Acceptance/Validation
Validate system on
environment as close
to production as
possible.
Live, immediate
feedback,
monitoring
Automation
Unit
API
Smoke
Smoke
Acceptance
API
UI Functional Regression
End2End
user focused tasks
Happy Path
Monitoring
verifying live
system
operation

5/25/2017
24
This whole process is ONLY possible with big,
significant, high-volume, fast, low maintenance,
excellent automation…with immediate feedback.
Not a little automation. A LOT!
Why are we talking about this with testers?
Special Cases-
“Functional tests teams” may take over:
• Performance testing
• Security testing
• Production Monitoring
• API/Web Service testing
• Testing in production
• Capacity Testing
• Platform Testing
• Device Compatibility
Or
• Our automated suites may be taken over by another team
or tool.
Solve issues with other types of tests

5/25/2017
25
Automated testing
Test Automation means Tool Suite
• UI tool/Functional
• API tool
• REST
• SOAP
• Security…
• Vulnerability
• Penetration
• Performance
• Cloud service for devices…whatever
• Monitoring in production
Not all tools work in different situations
The tool can’t get in the way.
A very popular browser tools has famously had problems
funning on multiple virtual machines at the same time.
The tool got in the way.
• If you are using a cloud solution- the tool has to work
• If you are using high numbers of VMs the tool has to
work.
• If you are testing in production, the tool cannot
interfere! Or worse, slow down or crash! (non-
intrusive)

5/25/2017
26
Special Topics
• SOA
• Regression and Automation
DevOps Overview

5/25/2017
27
SOA/MicroServices
Testing APIs,
Producing services
Using containers as service components on your
system.
Understanding how to test services- next technical
leap.
Your REST and SOAP testing, tools and automation
skills may need an upgrade.
MicroServices
How do you test?
Develop an API/Service Test Strategy
• Test in Isolation
• Test Integration
• Test in Scenario, E2E.
• Automate it.
MicroServices Test Strategy

5/25/2017
28
Regression Automation
What is regression testing?
And by now, it has to be Regression Automation.
Regression Testing
Regression testing is a big important thing.
Yet in many companies, it’s value is unclear.
Your regression suites in DevOps must be fast and give excellent
feedback to the teams on risks of speedy product delivery.
We will talk about what this means to automation in the next
section- but many teams have to rethink how they choose tests for
the regression suite for different places in Continuous Testing.

5/25/2017
29
Regression Testing
Description Time to execute Coverage Risk
Full
Core Functionality
High Priority/High Risk
Main Paths
Most Common Paths
Bug
Smoke Test
Financial Service
Consumer Product
US, Co-located teams.
“We are going to do DevOps.”
• Missing basic Agile principles. Missing modern software quality ideas.
• (Previous) Automation goal: automating every manual script is a good
thing.
• Over-focus on #s of test cases automated rather than what the test did.
• Low quality/old style test design.
Solution
• Train team on test case design, measuring coverage, choosing good
test data.
• Throw out existing useless automation suites.
• Begin better test design. Design intelligent meaningful tests.
• Define four sets of automated regression suites for different purposes,
environments and execution times.

5/25/2017
30
Regression Testing
Re-Think your regression strategy!
Does your regression suite have the main customer
flows? Long end2end tests or small isolated function
tests?
How old are your regression tests?
Do they find bugs?
Do they have value?
Regression Testing
You will probably need a few regression suites
CI- Smoke- small
Test- Some kind of “full”- perhaps bloated and old
Staging- task, customer scenario, Workflow- lean and
mean
Production/Monitoring- transaction- better not break!
Limited goals.

5/25/2017
31
DevOps Automated Testing
For some:
• Automated “full regression” suite can take days to run
• “Full regression” automated suites have gotten bloated and slow
• Once tests get into the regression suite, they never come out.
• Tests that got automated could get easily automated, not the
most important ones.
• Automated tests are small, isolated, nuclear. Not what users do.
With the “Agile on steroids” mentality, automation thought needs
to move from “bigger is better” to “lean and mean.”
A team may need to redesign, optimize or even cut the size of their
automated suites for continuous testing.
Automation:
• What tests are you automating?
– Full transaction, end to end, not only tiny isolated tests.
• Various suites
• For Instant feedback
• Flipping the triangle…or not
• Automation strategy
– Low maintenance framework, scalable
– Transferable
• Don’t be the constraint/bottleneck!
Continuous Testing

5/25/2017
32
Flipping the Triangle
Testing in Continuous Delivery
DevOps Automated Testing
In DevOps,
• feedback must be fast, immediate.
• Automated suites need to be groomed and cleaned up
regularly.
• Important tests only get into automated suites.
Automated tests need to:
• Validate.
• Find bugs.
• Mirror what users do.
• Assure consistency, monitor, feedback.
• The tests need to be economical.

5/25/2017
33
Business ERP software.
B2B Global, Not co-located teams.
3 years ago: “Lets Do DevOps!”
• Throw out all automation and start over.
– There is no nicer way to say it.
– Old style, intrusive tool.
• Old style automation- “over-checking” in every script.
• High maintenance tests.
• Design a suite of long, workflow, scenario customer focused
tests.
Automated Testing
Put together the thoughts on:
• What tests get run against what server
• The purpose of testing in a certain environment vs.
another
• What regression means in different cases.
• You may have 3 or 4 different automated suites run
regularly.
• Lean & Mean

5/25/2017
34
JumpStart and Summary
DevOps

5/25/2017
35
1. Get rid of the name “QA” – Everyone is responsible for testing and
quality. If your team has not fully grasped this value- you can never do
DevOps. Call them testers, test engineers, quality engineers- SDET-
whatever. They do not own quality.
2. Not enough fast, nimble, meaningful, current, easy-to-run automation.
3. No easy way to convey coverage: smoke test, full regression: code
coverage, happy path tests, main functionality, changed area regression,
bug regression, high priority regression – what are you doing and what is
left out to be fast and nimble. Never say “test everything” because you
don’t.
4. Teams forget about traditional, during-the-sprint, ad hoc and scenario
and forced error tests that may never be automated but have to be run
to see the new code works like a customer wants it – not what a unit test
validates.
5. Test environments, staging environments, testing in production – all
environment problems must be fixed. Made simple, fast and leverage
virtualization. Ops should help.
What are the problems/issues for testing?
6. Some test teams have never done performance or security testing
before. Particularly performance testing, in DevOps often gets run
along with functional regression suites- you may need to start
maintaining these also.
7. Data problems must be fixed; masking, randomizing, mirroring, current,
live– whatever – fix it and make it fast.
8. Deal with the risks of testing in production. Communicate the risks.
Don’t ignore it.
9. Automate, automate, automate- do it smart, fix it fast, unbreak it, do it
right (framework, organize, well-designed, thoughtful, fast-fast)
10. Your team may not realize the amount of collaboration needed to make
this work. Open, friendly, clear, all-for-one team communication is
mandatory.
11. You need easy, fast smoke tests in CI. Maybe even take over running
Jenkins/AutoBuild/Bamboo/whatever.
12. Learn about working in the Cloud and API/Service Testing, if you are
not already, you will be soon.
What are the problems for testing?

5/25/2017
36
1. Fix any Agile/Scrum problems
2. Get CI and Smoke Tests smooth, easy, great
3. Get the teams to better understand and agree to your use of “regression”
4. Pass all IT/Ops/ system problems over to other people- stop the
struggle. Get to know how the environments solution works.
5. Fill-in whatever knowledge gaps you find (containers, SOAP, Rest,
microservices)
6. Get fully knowledgeable of every testing task for your product regardless
of who does it- programmers, what testing happens in production?
– Gather info on security testing, performance testing, etc.
– On what platforms? Server? Mobile? Desktop?
7. Make sure you have an automation tool that very easily works in your
virtualized world and tool chain.
8. Make sure you have easy, fast, extensible automation framework.
9. Re-think how and what you automate: lean and mean.
10. Develop a deployment production line mentality: everything you do helps
or stops deployment.
Jumpstart
You have to know what you are talking about with DevOps;
what parts impact testing and what do not.
Most of DevOps is not about test teams.
Remember the Culture Shifts
Automate Lean and Mean
– Continuous testing does not mean continuously rerunning your
full regression.
– You may need MANY sets of regression suites
– These sets will be run at different times on different
environments.
– Your tools have to support this.
– Automated Testing keeps the pipeline moving
– Don’t be the bottleneck!
Summary

5/25/2017
37
Continuous Testing – Q&A
For more information on Continuous Testing Solutions
visit logigear.com .
Thank you!
To learn more about LogiGear’s training and QA consulting services email sales@logigear.com
Michael Hackett michaelh@logigear.com

Continuous Testing in DevOps

More Related Content

What's hot

Similar to Continuous Testing in DevOps

More from TechWell

Recently uploaded

Continuous Testing in DevOps