Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Continuous Testing
At the Speed of DevOps
Marc Hornbeek
Sr. Solutions Architect
2Spirent Communications www.spirent.com/solutions/devops
IDC December 2014
Fortune 1000 Survey
 Situation
 Average cost ...
3Spirent Communications www.spirent.com/solutions/devops
SVM
Build(s)
SUT
Commit
Commands
And
Responses
Source Code
P4, Gi...
4Spirent Communications www.spirent.com/solutions/devops
5Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Innovation
Test strategy requires criti...
6Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Quality
CT is more than just quality me...
7Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Time-to-market
Build CT for speed, and ...
8Spirent Communications www.spirent.com/solutions/devops
Continuous Testing means… Return-On-Investment
Penny wise,
pound ...
9Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practices
10Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #1 – Team and Culture
Architect...
11Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #2 – CT-Ready Tools
Restful API...
12Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #3 – Tools Integration
Plug-in
...
13Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #4 – Stability and Metrics
Pre-...
14Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #5 – CT Acceleration
Powerful s...
15Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #6 – CT Analytics
Test schedule...
16Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Best Practice #7 – Optimize CT Orchestration
...
17Spirent Communications www.spirent.com/solutions/devops
Continuous Testing Requires Expert Solutions
 Knowledge of CT b...
18Spirent Communications www.spirent.com/solutions/devops
Spirent CLEAR DevOps Solution
Continuous
Deployment
(CD)
Continu...
19Spirent Communications www.spirent.com/solutions/devops
Implementation Strategies – Non-Disruptive Measured Progress
Mac...
20Spirent Communications www.spirent.com/solutions/devops
Real World Example – Network Equipment Manufacturer
Metric Befor...
21Spirent Communications www.spirent.com/solutions/devops
Summary
 The hidden secret of DevOps
success is CT
 DevOps suc...
22Spirent Communications www.spirent.com/solutions/devops
spirent.com/solutions/devops
Upcoming SlideShare
Loading in …5
×

DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps

Continuous delivery is frightening to enterprise IT managers who see each new private, public or hybrid cloud infrastructure software change potentially causing service outages or security concerns.

This presentation by Marc Hornbeek, first shared at the DevOps Summit 2015 in London, explains Spirent’s comprehensive Clear DevOps Solution to support:
- Rapid paced continuous testing without compromising coverage or service quality
- Orchestration of service deployments over physical and virtual infrastructures
- Best practices for integrating continuous testing into CI infrastructures
- How to use continuous testing analytics for deployment decisions

  • Login to see the comments

DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps

  1. 1. Continuous Testing At the Speed of DevOps Marc Hornbeek Sr. Solutions Architect
  2. 2. 2Spirent Communications www.spirent.com/solutions/devops IDC December 2014 Fortune 1000 Survey  Situation  Average cost of failure: >$500K  Application development waste: 25%  Solution  83% using or evaluating DevOps  21.4 % investing in Testing/QA tools  Cautions  Failure rate when using current tools: 80%
  3. 3. 3Spirent Communications www.spirent.com/solutions/devops SVM Build(s) SUT Commit Commands And Responses Source Code P4, Git, SVN, etc. Deliver Dev, CI, QA Labs Physical, virtual, hybrid environments Test and Lab Management Checkout Images Test Info Results data Logs Response Info Test I/P Test O/P Images Analytics Dashboards, ALM, PE Not Ready Ready CI/CD Orchestration Tools ContinuousIntegrationCI Continuous Test CT ContinuousDeliveryCD Continuous Change Management CCMPre-Flight Development Artifact Repository Images, tests, configs, logs, results Software Changes
  4. 4. 4Spirent Communications www.spirent.com/solutions/devops
  5. 5. 5Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Innovation Test strategy requires critical thinking which enables innovation and efficient CT which reduces wasted corrective work, frees time for innovation.
  6. 6. 6Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Quality CT is more than just quality measurement, it is an active part of quality development, verification and deployment.
  7. 7. 7Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Time-to-market Build CT for speed, and don’t stop!
  8. 8. 8Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Return-On-Investment Penny wise, pound foolish CT is a major factor affecting Return-On Investment
  9. 9. 9Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practices
  10. 10. 10Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #1 – Team and Culture Architect CT Skills Test Design Reviews Workflow Analysis Collaboration Training
  11. 11. 11Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #2 – CT-Ready Tools Restful APIs Cache and pipeline Virtual and physical Vertical aggregation Test topologies Program agnostic Large scale
  12. 12. 12Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #3 – Tools Integration Plug-in CI Tools CT Tools Plug-in CD Tools Plug-in CM Tools Restful APIs Restful APIs Documented Reusable DevOps ready Regression tests
  13. 13. 13Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #4 – Stability and Metrics Pre-Flight Static analysis Unit tests Integration Feature tests System tests Regression STOP REVERT ALLOW
  14. 14. 14Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #5 – CT Acceleration Powerful servers Test design Pre-load Pre-configure Scale horizontally No waiting! Thresholds Aggregate results
  15. 15. 15Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #6 – CT Analytics Test schedules Test selection Resources Test phases CT Controls CI, CD Controls Code reverts Promotions Releases
  16. 16. 16Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #7 – Optimize CT Orchestration Orchestrating both physical and virtual test resources is important ! Physical Virtual Hybrid Catalogue Invoke Pipeline
  17. 17. 17Spirent Communications www.spirent.com/solutions/devops Continuous Testing Requires Expert Solutions  Knowledge of CT best practices  Cost of not getting it right  Delays due to higher Ops priorities  Ongoing cost of tool integrations, maintenance and enhancements 80% failure rate when using current tools IDC Fortune 1000 survey, December 2014
  18. 18. 18Spirent Communications www.spirent.com/solutions/devops Spirent CLEAR DevOps Solution Continuous Deployment (CD) Continuous Integration (CI) Plug-ins Plug-ins SUTTools Lab Management Physical, Virtual, Hybrid Lab Analytics ALM 6. Expertise and professional services 1. Test orchestration & lab management 2. Comprehensive suite of test tools 3. Physical, virtual and mixed hybrid labs 4. CI/CT/CD/CCM tools integration (EVCI) 5. CT analytics, ALM integration Orchestration Continuous Test (CT) CCM
  19. 19. 19Spirent Communications www.spirent.com/solutions/devops Implementation Strategies – Non-Disruptive Measured Progress Macro-Phases 1. Assessment to determine bottlenecks 2. Proof of concept 3. Horizontal integration 4. Vertical deployment Mini-Phases 1. Team & Culture 2. Tools integrations 3. Stabilize, measure 4. OptimizationsMacro- Phases Mini- Phases Initial Changing Optimized Micro-Phases 1. Change a little 2. Test 3. Deploy
  20. 20. 20Spirent Communications www.spirent.com/solutions/devops Real World Example – Network Equipment Manufacturer Metric Before After Major release (mo.) 6 3 Minor release (wks.) 4 2 # Features 113 150 Defects 1260 10 Integrations / day 0.5 100 Tests / day 0.3 10 Automated tests 5% 85% The primary contributing factor was CT !
  21. 21. 21Spirent Communications www.spirent.com/solutions/devops Summary  The hidden secret of DevOps success is CT  DevOps success will only be accomplished if CT is approached in accordance with best practices  Don't fall into the “do-CT- yourself” trap!
  22. 22. 22Spirent Communications www.spirent.com/solutions/devops spirent.com/solutions/devops

×