SlideShare a Scribd company logo
Continuous Test
Joan Wrabetz, CTO
October 2016
• Why is Test the last mile of DevOps
• The Challenge of Continuous Testing
• The Role of the Environment in Automating Testing
• Sandboxes for Automating Environments in DevOps
Agenda
A best practice that emphasizes the collaboration of
software developers & other information-technology
(IT) professionals while automating the process of
software delivery & infrastructure changes.
Businesses care because they require a methodology
to bridge product development and operations while
balancing time to market, product performance &
security.
WHAT IS DevOps?
Works in Dev
And then I run it here, right?
The problem is…
Or Worse Yet…
A best practice that emphasizes the collaboration of
software developers, Testers & other information-
technology (IT) professionals while automating the
process of software delivery & infrastructure changes.
Businesses care because they require a methodology
to bridge product development, testing, and
operations while balancing time to market, product
performance & security.
WHAT IS DevTestOps?
THE DevOps LIFECYCLE
Continuous Test
talk about this actually got this
DEVOPS @ ENTERPRISES
Performan
ce Security
Integration
UAT
My testing/staging environments need to
be similar to my _____________________ production environment
The challenge
 HYBRID/MIXED
 COMPLEX
 No Rollback
Continuous Integration
Applications are
containerized, virtualized,
API aware, and integrated
with DevOps
The environment that they
run on is not as pretty
Continuous Testing
Everything that you need in order to test your application (or infrastructure)
The Environment is…
Production
Data
Production
Data
(anonymous)
50 shades of Environments
Play it backwards
Application Under Test
Data Test Tools
3rd Party Services
Infrastructure
Data
50 shades of Environments
Play it backwards
D
t
DevOps Pipeline: Production
Data
Real Deployed
Infrastructure
Production
Database
50 shades of Environments
Play it backwards
D
t
DevOps Pipeline: Staging
Application Under Test
Data Test Tools
3rd Party Services
Infrastructure
Full
production
Data
(anonymous)
50 shades of Environments
Play it backwards
D
t
DevOps Pipeline: Test
Application Under Test
Data Test Tools
3rd Party Services
Infrastructure
Subset of
production
Data
(anonymous)
50 shades of Environments
Play it backwards
D
t
DevOps Pipeline: Development
Application Under Test
Data Test Tools
3rd Party Services
Infrastructure
Environments change throughout the DevOps Lifecycle
DevOps Pipeline
t
Dev
Application
Under Test
Data Test
Tools
3rd Party
Services
Infrastructure
Subse
t of
produ
ction
Data
(anon
ymou
s)
Application
Under Test
Data Test
Tools
3rd Party
Services
Infrastructure
D
t
Test
Application
Under Test
Data Test
Tools
3rd Party
Services
Infrastructur
e
Full
produc
tion
Data
(anony
mous)
Staging
Real Deployed
Infrastructure
Produc
tion
Databa
se
Production
Acceptance
Testing
Create Test
Environment
Deploy
Production
Config
App
Release
Automation
Cont.
Integration
Scan
Quality
Code
And Build
Operate and
Monitor
Integration
Testing
DEVOPS LIFECYCLE
CONTINUOUS INTEGRATION CONTINUOUS TESTING CONTINUOUS DEPLOYMENT
WHERE ENVIRONMENTS MATTER
Create Staging
Environment
A Sandbox is a container for
the environment where all
of the different
configurations can be
created
They are environment
centric:
They are flexible
They can be managed and
scheduled
They can be abstract
I need a Sandbox
They can be simple or complex
Enterprise Sandbox
My laptop…
Production…
• Model the infrastructure
• Describe applications
• Describe 3rd party services
• Enable data copy
• Automate setup
• Define processes
• Integrate with DevOps tools
Sandboxes are built with simple tools
DevOps + Containers + Sandboxes
Closing the DevOps Gap
MonitorOperateDeployReleaseTestBuildCodePlan
Sandboxes enable DevOps for
Infrastructure:
• Cloud Sandboxes
• Environment portability
• Environment consistency
• It is a context for keeping meta-data
• Automation
• Dynamic multi-tenancy for
Environments
Containers and Sandboxes
For DevOps Automation
Containers enable DevOps for
Applications:
• App portability
• Packaging
• It is a context for keeping meta-
data
• Automation Enabler
• Dynamic multi-tenancy for apps
A Cloud Sandbox Is…
“A Personal Replica of a Production
Environment for Automating DevOps”
Automate through
DevOps tools
Quali Cloud Sandboxes
Building Blocks for DevOps Automation
Personalized replicas of production environments
Reservation
Self-Service with Sophisticated Modeling and Automation
BI and
Analytics
Deploy
Anywhere
Set-up and
tear down
Environment
GUI or
API-driven
Connectivity
Applications and DataPhysical or Virtual Infrastructure
Orchestrate
DevOps and
BizOps
workflows
Security
“Full Stack”
On-Prem
or Hybrid
CloudMulti-tenant
Quali Cloud Shell – Industry –Leading Sandbox
29
Benefits of Sandboxes to DevOps
Faster Release Cycles
Lower TCO
Reduced Ops Failures
Automate
Config. Setup and teardown
Integrate with CI/CD Toolsets
Decrease Risk
Dev/Test in Production Replicas
Expand scope Eg. Security
Reduce Cost
Efficient Resource Utilization
Continuous Testing Best Practices
Separate environment handling from testing1
Map your sandbox grades2
Extend continuous integration to get to
continuous delivery
3
Successful pioneers like Netflix, Etsy and
Amazon took 5+ years to get to CD.
Q & A
• The Sandbox is Coming to DevOps
https://appdevelopermagazine.com/3584/2016/2/10/Cloud-Sandboxes-Are-Becoming-
the-Uber-Containers-for-the-DevOps-Cycle/
• The Surprising Truth About DevOps in Banks
https://www.infoq.com/articles/devops-in-banking
• Containers Enable DevOps and Hybrid Clouds, But Sandboxes Are Also Critical for IT
Success
http://datacenterpost.com/2015/11/containers-enable-devops-and-hybrid-clouds-but-
sandboxes-are-also-critical-for-it-success/
• DevOps, Hybrid Cloud & Sandboxes
http://www.networkcomputing.com/data-centers/devops-hybrid-cloud-
sandboxes/413832201
• Understanding DevOps – Part 4: Continuous Testing and Continuous Monitoring
https://sdarchitect.wordpress.com/2012/10/30/understanding-devops-part-4-
continuous-testing-and-continuous-monitoring/
References
33
Quali engages with its customers, partners and developers in many ways.
Stay in Touch
BlogsCommunity.quali.comwww.quali.com
Social Media
@Quali
@Qualisystems
Joan.w@Quali.com
@varbetz
34
www.quali.com
THANK YOU

More Related Content

What's hot

QA in DevOps: Transformation thru Automation via Jenkins
QA in DevOps:  Transformation thru Automation via JenkinsQA in DevOps:  Transformation thru Automation via Jenkins
QA in DevOps: Transformation thru Automation via JenkinsTatyana Kravtsov
 
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
Edureka!
 
2017 DevSecOps Survey
2017 DevSecOps Survey2017 DevSecOps Survey
2017 DevSecOps Survey
Sonatype
 
Keeping Your Continuous Test Automation Continuously Valuable
Keeping Your Continuous Test Automation Continuously ValuableKeeping Your Continuous Test Automation Continuously Valuable
Keeping Your Continuous Test Automation Continuously Valuable
Perfecto by Perforce
 
How do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about PipelinesHow do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about Pipelines
Thoughtworks
 
Sec devops 2.0 managing your robot army - final
Sec devops 2.0  managing your robot army - finalSec devops 2.0  managing your robot army - final
Sec devops 2.0 managing your robot army - final
Josh Bregman
 
Testing in DevOps world
Testing in DevOps worldTesting in DevOps world
Testing in DevOps world
Moataz Nabil
 
#ESPC19 How to do #DevOps with #SPFx
#ESPC19 How to do #DevOps with #SPFx#ESPC19 How to do #DevOps with #SPFx
#ESPC19 How to do #DevOps with #SPFx
Vincent Biret
 
Testing with Microsoft Technologies - Kick Off Session
Testing with Microsoft Technologies - Kick Off SessionTesting with Microsoft Technologies - Kick Off Session
Testing with Microsoft Technologies - Kick Off SessionMoataz Nabil
 
Increasing Quality with DevOps
Increasing Quality with DevOpsIncreasing Quality with DevOps
Increasing Quality with DevOps
Coveros, Inc.
 
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy EnvironmentsPete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
Peter Marshall
 
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
Simplilearn
 
MOPCON 2015 - Tips of Mobile Continuous Delivery
MOPCON 2015 - Tips of Mobile Continuous DeliveryMOPCON 2015 - Tips of Mobile Continuous Delivery
MOPCON 2015 - Tips of Mobile Continuous Delivery
anistar sung
 
Brave New World - A wider perspective of our opportunities
Brave New World - A wider perspective of our opportunitiesBrave New World - A wider perspective of our opportunities
Brave New World - A wider perspective of our opportunities
Jayathirtha Rao
 
AfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing IntroductionAfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing Introduction
Peter Marshall
 
Chris Northwood Camp Digital 21015 - Continuous delivery
Chris Northwood Camp Digital 21015 - Continuous deliveryChris Northwood Camp Digital 21015 - Continuous delivery
Chris Northwood Camp Digital 21015 - Continuous delivery
Nexer Digital
 
4 Testing Methods to Scale and Automate Your DevOps Pipeline
4 Testing Methods to Scale and Automate Your DevOps Pipeline4 Testing Methods to Scale and Automate Your DevOps Pipeline
4 Testing Methods to Scale and Automate Your DevOps Pipeline
Perfecto by Perforce
 
The Key to DevOps? Testing Early in the Pipeline
The Key to DevOps? Testing Early in the PipelineThe Key to DevOps? Testing Early in the Pipeline
The Key to DevOps? Testing Early in the Pipeline
DevOps.com
 
Continuous testing in agile projects 2015
Continuous testing in agile projects 2015Continuous testing in agile projects 2015
Continuous testing in agile projects 2015
Fabricio Epaminondas
 
Webinar: DevOps challenges facing QA
Webinar: DevOps challenges facing QAWebinar: DevOps challenges facing QA
Webinar: DevOps challenges facing QA
Qualitest
 

What's hot (20)

QA in DevOps: Transformation thru Automation via Jenkins
QA in DevOps:  Transformation thru Automation via JenkinsQA in DevOps:  Transformation thru Automation via Jenkins
QA in DevOps: Transformation thru Automation via Jenkins
 
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
Introduction to DevOps Tools | DevOps Training | DevOps Tutorial for Beginner...
 
2017 DevSecOps Survey
2017 DevSecOps Survey2017 DevSecOps Survey
2017 DevSecOps Survey
 
Keeping Your Continuous Test Automation Continuously Valuable
Keeping Your Continuous Test Automation Continuously ValuableKeeping Your Continuous Test Automation Continuously Valuable
Keeping Your Continuous Test Automation Continuously Valuable
 
How do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about PipelinesHow do you implement Continuous Delivery? Part 3: All about Pipelines
How do you implement Continuous Delivery? Part 3: All about Pipelines
 
Sec devops 2.0 managing your robot army - final
Sec devops 2.0  managing your robot army - finalSec devops 2.0  managing your robot army - final
Sec devops 2.0 managing your robot army - final
 
Testing in DevOps world
Testing in DevOps worldTesting in DevOps world
Testing in DevOps world
 
#ESPC19 How to do #DevOps with #SPFx
#ESPC19 How to do #DevOps with #SPFx#ESPC19 How to do #DevOps with #SPFx
#ESPC19 How to do #DevOps with #SPFx
 
Testing with Microsoft Technologies - Kick Off Session
Testing with Microsoft Technologies - Kick Off SessionTesting with Microsoft Technologies - Kick Off Session
Testing with Microsoft Technologies - Kick Off Session
 
Increasing Quality with DevOps
Increasing Quality with DevOpsIncreasing Quality with DevOps
Increasing Quality with DevOps
 
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy EnvironmentsPete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
Pete Marshall - casmadrid2015 - Continuous Delivery in Legacy Environments
 
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
DevOps Training | DevOps Training Video | DevOps Tools | DevOps Tutorial For ...
 
MOPCON 2015 - Tips of Mobile Continuous Delivery
MOPCON 2015 - Tips of Mobile Continuous DeliveryMOPCON 2015 - Tips of Mobile Continuous Delivery
MOPCON 2015 - Tips of Mobile Continuous Delivery
 
Brave New World - A wider perspective of our opportunities
Brave New World - A wider perspective of our opportunitiesBrave New World - A wider perspective of our opportunities
Brave New World - A wider perspective of our opportunities
 
AfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing IntroductionAfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing Introduction
 
Chris Northwood Camp Digital 21015 - Continuous delivery
Chris Northwood Camp Digital 21015 - Continuous deliveryChris Northwood Camp Digital 21015 - Continuous delivery
Chris Northwood Camp Digital 21015 - Continuous delivery
 
4 Testing Methods to Scale and Automate Your DevOps Pipeline
4 Testing Methods to Scale and Automate Your DevOps Pipeline4 Testing Methods to Scale and Automate Your DevOps Pipeline
4 Testing Methods to Scale and Automate Your DevOps Pipeline
 
The Key to DevOps? Testing Early in the Pipeline
The Key to DevOps? Testing Early in the PipelineThe Key to DevOps? Testing Early in the Pipeline
The Key to DevOps? Testing Early in the Pipeline
 
Continuous testing in agile projects 2015
Continuous testing in agile projects 2015Continuous testing in agile projects 2015
Continuous testing in agile projects 2015
 
Webinar: DevOps challenges facing QA
Webinar: DevOps challenges facing QAWebinar: DevOps challenges facing QA
Webinar: DevOps challenges facing QA
 

Similar to 2016 quali continuous testing quest for quality conference

Continuous Delivery with a PaaS Application
Continuous Delivery with a PaaS ApplicationContinuous Delivery with a PaaS Application
Continuous Delivery with a PaaS Application
Mark Rendell
 
The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017
Micro Focus
 
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
minseok kim
 
DevOps on AWS
DevOps on AWSDevOps on AWS
DevOps on AWS
Amazon Web Services
 
DevOps and Application Delivery for Hybrid Cloud - DevOpsSummit session
DevOps and Application Delivery for Hybrid Cloud  - DevOpsSummit sessionDevOps and Application Delivery for Hybrid Cloud  - DevOpsSummit session
DevOps and Application Delivery for Hybrid Cloud - DevOpsSummit session
Sanjeev Sharma
 
Dev ops
Dev opsDev ops
Dev ops
Tomy Rhymond
 
Dops 1033 dev-ops_review_final
Dops 1033 dev-ops_review_finalDops 1033 dev-ops_review_final
Dops 1033 dev-ops_review_finalDaniel Berg
 
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform EnvironmentsDeployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
IBM UrbanCode Products
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"
Stephanie Locke
 
Visual Studio ALM and DevOps Tools Walkthrough
Visual Studio ALM and DevOps Tools WalkthroughVisual Studio ALM and DevOps Tools Walkthrough
Visual Studio ALM and DevOps Tools Walkthrough
Angela Dugan
 
Managing IT environment complexity in a Multi-Cloud World
Managing IT environment complexity in a Multi-Cloud WorldManaging IT environment complexity in a Multi-Cloud World
Managing IT environment complexity in a Multi-Cloud World
Shashi Kiran
 
Implementing Azure DevOps with your Testing Project
Implementing Azure DevOps with your Testing ProjectImplementing Azure DevOps with your Testing Project
Implementing Azure DevOps with your Testing Project
RTTS
 
Continuous Delivery in the Enterprise
Continuous Delivery in the EnterpriseContinuous Delivery in the Enterprise
Continuous Delivery in the Enterprise
IBM UrbanCode Products
 
Enterprise DevOps: Scaling Build, Deploy, Test, Release
Enterprise DevOps: Scaling Build, Deploy, Test, ReleaseEnterprise DevOps: Scaling Build, Deploy, Test, Release
Enterprise DevOps: Scaling Build, Deploy, Test, Release
IBM UrbanCode Products
 
DTS-1778 Understanding DevOps - IBM InterConnect Session
DTS-1778 Understanding DevOps - IBM InterConnect SessionDTS-1778 Understanding DevOps - IBM InterConnect Session
DTS-1778 Understanding DevOps - IBM InterConnect Session
Sanjeev Sharma
 
Dev ops in agile - 1st Conference Melbourne
Dev ops in agile - 1st Conference MelbourneDev ops in agile - 1st Conference Melbourne
Dev ops in agile - 1st Conference Melbourne
Mirco Hering
 
Continuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hallContinuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hall
Peter Marshall
 
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
Siva Rama Krishna Chunduru
 
How to Add Perfecto to Your CI
How to Add Perfecto to Your CIHow to Add Perfecto to Your CI
How to Add Perfecto to Your CI
Lizzy Guido (she/her)
 
No Devops Without Continuous Testing
No Devops Without Continuous TestingNo Devops Without Continuous Testing
No Devops Without Continuous Testing
Parasoft
 

Similar to 2016 quali continuous testing quest for quality conference (20)

Continuous Delivery with a PaaS Application
Continuous Delivery with a PaaS ApplicationContinuous Delivery with a PaaS Application
Continuous Delivery with a PaaS Application
 
The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017
 
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
Pivotal korea transformation_strategy_seminar_enterprise_dev_ops_20160630_v1.0
 
DevOps on AWS
DevOps on AWSDevOps on AWS
DevOps on AWS
 
DevOps and Application Delivery for Hybrid Cloud - DevOpsSummit session
DevOps and Application Delivery for Hybrid Cloud  - DevOpsSummit sessionDevOps and Application Delivery for Hybrid Cloud  - DevOpsSummit session
DevOps and Application Delivery for Hybrid Cloud - DevOpsSummit session
 
Dev ops
Dev opsDev ops
Dev ops
 
Dops 1033 dev-ops_review_final
Dops 1033 dev-ops_review_finalDops 1033 dev-ops_review_final
Dops 1033 dev-ops_review_final
 
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform EnvironmentsDeployment Automation for Hybrid Cloud and Multi-Platform Environments
Deployment Automation for Hybrid Cloud and Multi-Platform Environments
 
Let's banish "it works on my machine"
Let's banish "it works on my machine"Let's banish "it works on my machine"
Let's banish "it works on my machine"
 
Visual Studio ALM and DevOps Tools Walkthrough
Visual Studio ALM and DevOps Tools WalkthroughVisual Studio ALM and DevOps Tools Walkthrough
Visual Studio ALM and DevOps Tools Walkthrough
 
Managing IT environment complexity in a Multi-Cloud World
Managing IT environment complexity in a Multi-Cloud WorldManaging IT environment complexity in a Multi-Cloud World
Managing IT environment complexity in a Multi-Cloud World
 
Implementing Azure DevOps with your Testing Project
Implementing Azure DevOps with your Testing ProjectImplementing Azure DevOps with your Testing Project
Implementing Azure DevOps with your Testing Project
 
Continuous Delivery in the Enterprise
Continuous Delivery in the EnterpriseContinuous Delivery in the Enterprise
Continuous Delivery in the Enterprise
 
Enterprise DevOps: Scaling Build, Deploy, Test, Release
Enterprise DevOps: Scaling Build, Deploy, Test, ReleaseEnterprise DevOps: Scaling Build, Deploy, Test, Release
Enterprise DevOps: Scaling Build, Deploy, Test, Release
 
DTS-1778 Understanding DevOps - IBM InterConnect Session
DTS-1778 Understanding DevOps - IBM InterConnect SessionDTS-1778 Understanding DevOps - IBM InterConnect Session
DTS-1778 Understanding DevOps - IBM InterConnect Session
 
Dev ops in agile - 1st Conference Melbourne
Dev ops in agile - 1st Conference MelbourneDev ops in agile - 1st Conference Melbourne
Dev ops in agile - 1st Conference Melbourne
 
Continuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hallContinuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hall
 
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
DevOps For Everyone: Bringing DevOps Success to Every App and Every Role in y...
 
How to Add Perfecto to Your CI
How to Add Perfecto to Your CIHow to Add Perfecto to Your CI
How to Add Perfecto to Your CI
 
No Devops Without Continuous Testing
No Devops Without Continuous TestingNo Devops Without Continuous Testing
No Devops Without Continuous Testing
 

More from QualiQuali

Accelerating Network Transformation with Self-service Environments and Orches...
Accelerating Network Transformation with Self-service Environments and Orches...Accelerating Network Transformation with Self-service Environments and Orches...
Accelerating Network Transformation with Self-service Environments and Orches...
QualiQuali
 
GDPR Compliance Countdown - Is your Application environment ready?
GDPR Compliance Countdown - Is your Application environment ready?GDPR Compliance Countdown - Is your Application environment ready?
GDPR Compliance Countdown - Is your Application environment ready?
QualiQuali
 
implanting DevOps at scale using dynamic test environments
implanting DevOps at scale using dynamic test environmentsimplanting DevOps at scale using dynamic test environments
implanting DevOps at scale using dynamic test environments
QualiQuali
 
The holy grail of automated labs self-service with l1 switching
The holy grail of automated labs   self-service with l1 switchingThe holy grail of automated labs   self-service with l1 switching
The holy grail of automated labs self-service with l1 switching
QualiQuali
 
Continuous testing webinar 041017 slideshare
Continuous testing webinar 041017 slideshareContinuous testing webinar 041017 slideshare
Continuous testing webinar 041017 slideshare
QualiQuali
 
Cloud Sandboxing Solution for Cyber Ranges
Cloud Sandboxing Solution for Cyber RangesCloud Sandboxing Solution for Cyber Ranges
Cloud Sandboxing Solution for Cyber Ranges
QualiQuali
 
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frogWebinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
QualiQuali
 
2016 quali dev ops_delivery of things_conference
2016 quali dev ops_delivery of things_conference2016 quali dev ops_delivery of things_conference
2016 quali dev ops_delivery of things_conference
QualiQuali
 
Quali webinar de-mystifyind dev_ops-a practitioner’s perspective
Quali webinar de-mystifyind dev_ops-a practitioner’s perspectiveQuali webinar de-mystifyind dev_ops-a practitioner’s perspective
Quali webinar de-mystifyind dev_ops-a practitioner’s perspective
QualiQuali
 
Mind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps ChasmMind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps Chasm
QualiQuali
 
Mind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps ChasmMind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps Chasm
QualiQuali
 
Heterogeneous Self-Service Automation for SDN Dev/Test
Heterogeneous Self-Service Automation for SDN Dev/TestHeterogeneous Self-Service Automation for SDN Dev/Test
Heterogeneous Self-Service Automation for SDN Dev/TestQualiQuali
 

More from QualiQuali (12)

Accelerating Network Transformation with Self-service Environments and Orches...
Accelerating Network Transformation with Self-service Environments and Orches...Accelerating Network Transformation with Self-service Environments and Orches...
Accelerating Network Transformation with Self-service Environments and Orches...
 
GDPR Compliance Countdown - Is your Application environment ready?
GDPR Compliance Countdown - Is your Application environment ready?GDPR Compliance Countdown - Is your Application environment ready?
GDPR Compliance Countdown - Is your Application environment ready?
 
implanting DevOps at scale using dynamic test environments
implanting DevOps at scale using dynamic test environmentsimplanting DevOps at scale using dynamic test environments
implanting DevOps at scale using dynamic test environments
 
The holy grail of automated labs self-service with l1 switching
The holy grail of automated labs   self-service with l1 switchingThe holy grail of automated labs   self-service with l1 switching
The holy grail of automated labs self-service with l1 switching
 
Continuous testing webinar 041017 slideshare
Continuous testing webinar 041017 slideshareContinuous testing webinar 041017 slideshare
Continuous testing webinar 041017 slideshare
 
Cloud Sandboxing Solution for Cyber Ranges
Cloud Sandboxing Solution for Cyber RangesCloud Sandboxing Solution for Cyber Ranges
Cloud Sandboxing Solution for Cyber Ranges
 
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frogWebinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
Webinar leveraging-cloud-sandboxes-with-ansible-jenkins-j frog
 
2016 quali dev ops_delivery of things_conference
2016 quali dev ops_delivery of things_conference2016 quali dev ops_delivery of things_conference
2016 quali dev ops_delivery of things_conference
 
Quali webinar de-mystifyind dev_ops-a practitioner’s perspective
Quali webinar de-mystifyind dev_ops-a practitioner’s perspectiveQuali webinar de-mystifyind dev_ops-a practitioner’s perspective
Quali webinar de-mystifyind dev_ops-a practitioner’s perspective
 
Mind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps ChasmMind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps Chasm
 
Mind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps ChasmMind the Gap: Crossing the DevOps Chasm
Mind the Gap: Crossing the DevOps Chasm
 
Heterogeneous Self-Service Automation for SDN Dev/Test
Heterogeneous Self-Service Automation for SDN Dev/TestHeterogeneous Self-Service Automation for SDN Dev/Test
Heterogeneous Self-Service Automation for SDN Dev/Test
 

Recently uploaded

ML for identifying fraud using open blockchain data.pptx
ML for identifying fraud using open blockchain data.pptxML for identifying fraud using open blockchain data.pptx
ML for identifying fraud using open blockchain data.pptx
Vijay Dialani, PhD
 
WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234
AafreenAbuthahir2
 
Planning Of Procurement o different goods and services
Planning Of Procurement o different goods and servicesPlanning Of Procurement o different goods and services
Planning Of Procurement o different goods and services
JoytuBarua2
 
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Dr.Costas Sachpazis
 
Railway Signalling Principles Edition 3.pdf
Railway Signalling Principles Edition 3.pdfRailway Signalling Principles Edition 3.pdf
Railway Signalling Principles Edition 3.pdf
TeeVichai
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
R&R Consult
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
Kamal Acharya
 
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
bakpo1
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
ankuprajapati0525
 
ethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.pptethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.ppt
Jayaprasanna4
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
thanhdowork
 
road safety engineering r s e unit 3.pdf
road safety engineering  r s e unit 3.pdfroad safety engineering  r s e unit 3.pdf
road safety engineering r s e unit 3.pdf
VENKATESHvenky89705
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
MdTanvirMahtab2
 
H.Seo, ICLR 2024, MLILAB, KAIST AI.pdf
H.Seo,  ICLR 2024, MLILAB,  KAIST AI.pdfH.Seo,  ICLR 2024, MLILAB,  KAIST AI.pdf
H.Seo, ICLR 2024, MLILAB, KAIST AI.pdf
MLILAB
 
HYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generationHYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generation
Robbie Edward Sayers
 
Runway Orientation Based on the Wind Rose Diagram.pptx
Runway Orientation Based on the Wind Rose Diagram.pptxRunway Orientation Based on the Wind Rose Diagram.pptx
Runway Orientation Based on the Wind Rose Diagram.pptx
SupreethSP4
 
block diagram and signal flow graph representation
block diagram and signal flow graph representationblock diagram and signal flow graph representation
block diagram and signal flow graph representation
Divya Somashekar
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
fxintegritypublishin
 
Gen AI Study Jams _ For the GDSC Leads in India.pdf
Gen AI Study Jams _ For the GDSC Leads in India.pdfGen AI Study Jams _ For the GDSC Leads in India.pdf
Gen AI Study Jams _ For the GDSC Leads in India.pdf
gdsczhcet
 
Architectural Portfolio Sean Lockwood
Architectural Portfolio Sean LockwoodArchitectural Portfolio Sean Lockwood
Architectural Portfolio Sean Lockwood
seandesed
 

Recently uploaded (20)

ML for identifying fraud using open blockchain data.pptx
ML for identifying fraud using open blockchain data.pptxML for identifying fraud using open blockchain data.pptx
ML for identifying fraud using open blockchain data.pptx
 
WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234WATER CRISIS and its solutions-pptx 1234
WATER CRISIS and its solutions-pptx 1234
 
Planning Of Procurement o different goods and services
Planning Of Procurement o different goods and servicesPlanning Of Procurement o different goods and services
Planning Of Procurement o different goods and services
 
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
 
Railway Signalling Principles Edition 3.pdf
Railway Signalling Principles Edition 3.pdfRailway Signalling Principles Edition 3.pdf
Railway Signalling Principles Edition 3.pdf
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
 
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
一比一原版(SFU毕业证)西蒙菲莎大学毕业证成绩单如何办理
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
 
ethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.pptethical hacking-mobile hacking methods.ppt
ethical hacking-mobile hacking methods.ppt
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
 
road safety engineering r s e unit 3.pdf
road safety engineering  r s e unit 3.pdfroad safety engineering  r s e unit 3.pdf
road safety engineering r s e unit 3.pdf
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
 
H.Seo, ICLR 2024, MLILAB, KAIST AI.pdf
H.Seo,  ICLR 2024, MLILAB,  KAIST AI.pdfH.Seo,  ICLR 2024, MLILAB,  KAIST AI.pdf
H.Seo, ICLR 2024, MLILAB, KAIST AI.pdf
 
HYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generationHYDROPOWER - Hydroelectric power generation
HYDROPOWER - Hydroelectric power generation
 
Runway Orientation Based on the Wind Rose Diagram.pptx
Runway Orientation Based on the Wind Rose Diagram.pptxRunway Orientation Based on the Wind Rose Diagram.pptx
Runway Orientation Based on the Wind Rose Diagram.pptx
 
block diagram and signal flow graph representation
block diagram and signal flow graph representationblock diagram and signal flow graph representation
block diagram and signal flow graph representation
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
 
Gen AI Study Jams _ For the GDSC Leads in India.pdf
Gen AI Study Jams _ For the GDSC Leads in India.pdfGen AI Study Jams _ For the GDSC Leads in India.pdf
Gen AI Study Jams _ For the GDSC Leads in India.pdf
 
Architectural Portfolio Sean Lockwood
Architectural Portfolio Sean LockwoodArchitectural Portfolio Sean Lockwood
Architectural Portfolio Sean Lockwood
 

2016 quali continuous testing quest for quality conference

  • 1. Continuous Test Joan Wrabetz, CTO October 2016
  • 2. • Why is Test the last mile of DevOps • The Challenge of Continuous Testing • The Role of the Environment in Automating Testing • Sandboxes for Automating Environments in DevOps Agenda
  • 3. A best practice that emphasizes the collaboration of software developers & other information-technology (IT) professionals while automating the process of software delivery & infrastructure changes. Businesses care because they require a methodology to bridge product development and operations while balancing time to market, product performance & security. WHAT IS DevOps?
  • 5. And then I run it here, right?
  • 8. A best practice that emphasizes the collaboration of software developers, Testers & other information- technology (IT) professionals while automating the process of software delivery & infrastructure changes. Businesses care because they require a methodology to bridge product development, testing, and operations while balancing time to market, product performance & security. WHAT IS DevTestOps?
  • 10. talk about this actually got this DEVOPS @ ENTERPRISES Performan ce Security Integration UAT
  • 11. My testing/staging environments need to be similar to my _____________________ production environment The challenge  HYBRID/MIXED  COMPLEX  No Rollback
  • 12. Continuous Integration Applications are containerized, virtualized, API aware, and integrated with DevOps The environment that they run on is not as pretty Continuous Testing
  • 13. Everything that you need in order to test your application (or infrastructure) The Environment is… Production Data Production Data (anonymous)
  • 14. 50 shades of Environments Play it backwards Application Under Test Data Test Tools 3rd Party Services Infrastructure Data
  • 15. 50 shades of Environments Play it backwards D t DevOps Pipeline: Production Data Real Deployed Infrastructure Production Database
  • 16. 50 shades of Environments Play it backwards D t DevOps Pipeline: Staging Application Under Test Data Test Tools 3rd Party Services Infrastructure Full production Data (anonymous)
  • 17. 50 shades of Environments Play it backwards D t DevOps Pipeline: Test Application Under Test Data Test Tools 3rd Party Services Infrastructure Subset of production Data (anonymous)
  • 18. 50 shades of Environments Play it backwards D t DevOps Pipeline: Development Application Under Test Data Test Tools 3rd Party Services Infrastructure
  • 19. Environments change throughout the DevOps Lifecycle DevOps Pipeline t Dev Application Under Test Data Test Tools 3rd Party Services Infrastructure Subse t of produ ction Data (anon ymou s) Application Under Test Data Test Tools 3rd Party Services Infrastructure D t Test Application Under Test Data Test Tools 3rd Party Services Infrastructur e Full produc tion Data (anony mous) Staging Real Deployed Infrastructure Produc tion Databa se Production
  • 20. Acceptance Testing Create Test Environment Deploy Production Config App Release Automation Cont. Integration Scan Quality Code And Build Operate and Monitor Integration Testing DEVOPS LIFECYCLE CONTINUOUS INTEGRATION CONTINUOUS TESTING CONTINUOUS DEPLOYMENT WHERE ENVIRONMENTS MATTER Create Staging Environment
  • 21. A Sandbox is a container for the environment where all of the different configurations can be created They are environment centric: They are flexible They can be managed and scheduled They can be abstract I need a Sandbox
  • 22. They can be simple or complex Enterprise Sandbox My laptop… Production…
  • 23. • Model the infrastructure • Describe applications • Describe 3rd party services • Enable data copy • Automate setup • Define processes • Integrate with DevOps tools Sandboxes are built with simple tools
  • 24. DevOps + Containers + Sandboxes Closing the DevOps Gap MonitorOperateDeployReleaseTestBuildCodePlan
  • 25. Sandboxes enable DevOps for Infrastructure: • Cloud Sandboxes • Environment portability • Environment consistency • It is a context for keeping meta-data • Automation • Dynamic multi-tenancy for Environments Containers and Sandboxes For DevOps Automation Containers enable DevOps for Applications: • App portability • Packaging • It is a context for keeping meta- data • Automation Enabler • Dynamic multi-tenancy for apps
  • 26. A Cloud Sandbox Is… “A Personal Replica of a Production Environment for Automating DevOps” Automate through DevOps tools
  • 27. Quali Cloud Sandboxes Building Blocks for DevOps Automation Personalized replicas of production environments Reservation Self-Service with Sophisticated Modeling and Automation BI and Analytics Deploy Anywhere Set-up and tear down Environment GUI or API-driven Connectivity Applications and DataPhysical or Virtual Infrastructure Orchestrate DevOps and BizOps workflows Security “Full Stack” On-Prem or Hybrid CloudMulti-tenant
  • 28. Quali Cloud Shell – Industry –Leading Sandbox
  • 29. 29 Benefits of Sandboxes to DevOps Faster Release Cycles Lower TCO Reduced Ops Failures Automate Config. Setup and teardown Integrate with CI/CD Toolsets Decrease Risk Dev/Test in Production Replicas Expand scope Eg. Security Reduce Cost Efficient Resource Utilization
  • 30. Continuous Testing Best Practices Separate environment handling from testing1 Map your sandbox grades2 Extend continuous integration to get to continuous delivery 3 Successful pioneers like Netflix, Etsy and Amazon took 5+ years to get to CD.
  • 31. Q & A
  • 32. • The Sandbox is Coming to DevOps https://appdevelopermagazine.com/3584/2016/2/10/Cloud-Sandboxes-Are-Becoming- the-Uber-Containers-for-the-DevOps-Cycle/ • The Surprising Truth About DevOps in Banks https://www.infoq.com/articles/devops-in-banking • Containers Enable DevOps and Hybrid Clouds, But Sandboxes Are Also Critical for IT Success http://datacenterpost.com/2015/11/containers-enable-devops-and-hybrid-clouds-but- sandboxes-are-also-critical-for-it-success/ • DevOps, Hybrid Cloud & Sandboxes http://www.networkcomputing.com/data-centers/devops-hybrid-cloud- sandboxes/413832201 • Understanding DevOps – Part 4: Continuous Testing and Continuous Monitoring https://sdarchitect.wordpress.com/2012/10/30/understanding-devops-part-4- continuous-testing-and-continuous-monitoring/ References
  • 33. 33 Quali engages with its customers, partners and developers in many ways. Stay in Touch BlogsCommunity.quali.comwww.quali.com Social Media @Quali @Qualisystems Joan.w@Quali.com @varbetz

Editor's Notes

  1. Let’s look at the definition. It focuses on the relationship between developers and IT Operations. It also focuses on the bridging of product development and operations. If we were to take this literally, what kinds of problems would ensue?
  2. From the developers perspective, this means that once it works in development…
  3. …We can deploy it in production
  4. Because of the focus on development and operations, some people interpret DevOps to mean that test can be eliminated.
  5. So, let’s try to re-make the definition of DevOps to be more clear…
  6. Developers want to focus on what they do best: develop Operations wants to focus on the reliable operation of the production environment. Can we really bypass test, staging, etc., and move code rapidly into the production environment? We can move rapidly and even automatically, through the dev and test process. But, eliminating test is not the solution.
  7. So, what is the state of testing today? Well, it’s not going too well. Why?
  8. The reason is often the testing or staging environments. When we talk about integration, security, performance testing – they REALLY need to mimic production environment, otherwise we could seriously risk quality and security. And this is specifically painful in domains where rollback in production is not a viable option, and the mean time to recovery has to be close to zero in order to be tolerated, it’s critical not to miss anything in the testing and staging phases. Which makes things more challenging and slower – it’s impossible to cut corners. The environments are many times hybrid or mixed. It could be legacy systems, a mix of on-prem and off-prem infrastructure. The environments could be very complex. A part of the environment is also the test equipment or simulators, control network configuration, use high availability or distributed architecture. The data needs to be production data or close to it. You need to actually have all the different components.
  9. Significant changes have helped to make applications amenable to DevOps and automation. Apps are virtualized and containerized. They are API aware and can be more readily run in different environments without changes. API awareness also enables applications to be installed and run through automation. This is not to be underestimated in enabling continuous integration / continuous testing or continuous Delivery. But, the environment must also be created as part of continuous testing and as noted earlier, this environment gets more complex and more critical as you move through testing and staging. Yet, very little has been done to make the environment as portable and re-producable as applications. The environmennt would need to be portable and easily packaged and also automatically re-producable.
  10. So, let’s look in more detail at what is in the environment and how it changes throughout the DevOps cycle. In development, the environment may just contain the application under test, represented here by the box. The environment also needs to contain the operating environment in which the application runs, or at least a minimum subset of it. For example, if our developer is changing a webclient, he or she just needs a browser in order to test it. But, as the DevOps train moves into large scale testing of various types – integration, regression, performance, and security testing. It may require some production data, some additional tools and services. We are in a disruptive time when the production environment is also changing – becoming more hybrid or moving from on-premise to public cloud. This also needs to be reflected during testing. As we move beyond development and unit testing, into regression testing and integration testing, the environment gets more complicated. We still have the application under test. In addition, the operating environment is a test lab which is a shared lab. It usually contains some components that are closer to production, like the network environment or the capability for testing in a mobile environment. In addition, the test tools may need to be part of the environment. When test includes end to end testing and performance testing, th
  11. So, let’s look at the categories of things that make up the environment. First, we have the application under test. It always needs to be there. We also need the infrastructure (physical, virtual, cloud, network) that the application will run on. It is important to note that the infrastructure available to developers and testers is usually quite different than the infrastructure that will be used in production. We may need some or all of production data that applications use. This is a big sticking point because incorporating this data into dev and test means protecting privacy laws by anonymizing the data in many verticals. How many of you in the audience, test using some or all of production data? Then, there are many additional services that the application depends on when it runs in production. Including the mobile network and mobile devices, and many other services. There is a whole practice around service virtualization within test tools in order to provide ways to test without the full service environment. There is also a requirement to mimic the production traffic as well and especially for performance and security testing, where the traffic in production really needs to be simulated in order to accurately test.
  12. Let’s start in production. What needs to be there? Well, the application under test will be there in production, along with the actual operating infrastructure and all of the 3rd party services that the application depends on. And, of course the production data sources are there.
  13. Now backing up to the process of staging, how much of production needs to be replicated? Well, as much as possible. So, there will be a full complement of virtual services and probably some traffic generation to simulate the production activity. In addition, if possible it would be desirable to have a full copy of the production databases. There may be some test tools in use to validate performance on the infrastructure. Usually, the infrastructure closely matches the production infrastructure during staging.
  14. Backing up to the testing phase, the infrastructure used might be more limited and perhaps a full copy of the production datasources may not be required. But the requirement to protect the privacy of that data becomes more critical because a much wider group of people have access to test environments. In addition, many of the 3rd party services are replaced by virtual services to enable reproducable and automatable testing.
  15. As we move back to development, the teams are focused on development and unit test. So, the tools in the environment are really mostly development tools, IDEs, and of course, the application under test. The application under test might be fairly complex, involving multiple components and even network segments, but often most of this is ignored. What is most telling about the story playing it backwards is that we can see that development environments simply bear little resemblance to production. This means that skipping from development all the way to production can be very dangerous.
  16. To summarize, the environment changes pretty dramatically between development and production. Solving this problem is critical to successfully implementing continuous delivery.
  17. The key to DevOps is continuous testing. The key to succeeding with continuous testing depends heavily on creating a more sophisticated environment for the application to be tested in.
  18. What is needed to solving the environments problem is to create sandboxes.
  19. A sandbox is malleable and can be made to reproduce any environment from simple to complex, as needed for the stage of the DevOps cycle.
  20. Of course, sandboxes need to come with tools for building these different environments, including all of the components that we have discussed. For example, how do you re-create a production database? The tools need to enable making a safe copy of production databases. And of course, sandboxes need to be fully automated, or they cannot smoothly participate in a DevOps process. And, they need to work frictionlessly with DevOps tools
  21. Let’s view this from the perspective of Devops. The Dev team will first implement continuous integration to automate the creation of an application under test package – taking advantage or virtualization or containers. The CI tools will trigger testing of the application. If we add sandboxes, the CI tool can automate the packaging of the application and the creation of the sandbox that pulls together all of the components of the environment. And, the process can be identical regardless of how the sandbox internals change.
  22. Quick analogy. Sandboxes enable DevOps for infrastructure in a similar way to how Containers enable devops for applications.
  23. So, what is a sandbox, really. Well, it is all about creating the environment automatically.Remembering that the environment includes the infrastructure, tools, data, services and the application under test. We need to be able to create the infrastructure that is required – physical, virtual, cloud and network. We need to deploy virtual services (which are types of applications) We need to deploy the application under test In order to facilitate automation – required for DevOps we need to support orchestration. For example, automatically set up the environment and automatically tear it down. Automatically get inputs and produce outputs, etc. And, finally, the sandbox must offer a way to be driven by DevOps tools – this is done by having a set of APIs for initiating and controlling sandboxes. With all of these capabilities, a sandbox can re-create production and do it within the DevOps process.
  24. This means that sandboxes have capabilities to support these activities: 1) automatically describing and creating the infrastructure (this is infrastructure as code + modeling) 2) Providing application templates and data copies 3) orchestrating activities in the sandbox 4) being driven by APIs
  25. Example: Security testing sandbox. Build1: The application under test is a typical web application with a web server, some forms that connect to email, and a database with some of the data in it. In security testing, we are testing whether the application creates any new security holes. We will call this the “white team”. Build 2: We have a separate setup in the sandbox for an application that generates specific scenarios for security breaches. These can be automated and the sandbox provides a separate subnet where these can be run from along with a couple of VMs for putting the code on. Build 3: We have a traffic generator (in this case its virtual) that can either generate background traffic that is typical for production, or it can also re-play traffic that includes a breach. Build 4: We have some tools for detecting and analyzing breaches which are helpful for the testing process or the debugging process. Build 5: Finally, we have the network setup that allows each group to be on their own subnet created from VLANS and also incorporates some firewalls and a central switch. There is built in orchestration for common test runs, such as : 0) set up this sandbox 1) start the traffic pattern 2) verify that the breach is in progress (using the sniffer) and 3) analyze the results of the breach using the malware analyzer 4) stop the traffic and 5) shutdown the sandbox. This sandbox could be initiated by a tool such as Jenkins or the CloudBees pipeline tool.
  26. Separate environment handling from testing Environment management is a full time job (DevOps team) Environment automation is not the same as test automation (and it can also be used by manual testers. And other operations – like support or partnering with 3rd parties for development and testing) Map your sandbox grades Dev environment, Test environment, Staging environment. They don’t all have to be production clones. What is the minimal environment that each of them requires? Extend continuous integration to get to continuous delivery Baby steps Definition of Done - automation is only done when it includes everything you need to run