DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
Report
Share
•8 likes•8,271 views
DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
•8 likes•8,271 views
Report
Share
Download to read offline
Continuous delivery is frightening to enterprise IT managers who see each new private, public or hybrid cloud infrastructure software change potentially causing service outages or security concerns. This presentation by Marc Hornbeek, first shared at the DevOps Summit 2015 in London, explains Spirent’s comprehensive Clear DevOps Solution to support: - Rapid paced continuous testing without compromising coverage or service quality - Orchestration of service deployments over physical and virtual infrastructures - Best practices for integrating continuous testing into CI infrastructures - How to use continuous testing analytics for deployment decisions
Recommended
More Related Content
What's hot
What's hot(20)
Similar to DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
Similar to DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps(20)
More from Sailaja Tennati
More from Sailaja Tennati(19)
Recently uploaded
Recently uploaded(20)
DevOps Summit 2015 Presentation: Continuous Testing At the Speed of DevOps
- 1. Continuous Testing At the Speed of DevOps Marc Hornbeek Sr. Solutions Architect
- 2. 2Spirent Communications www.spirent.com/solutions/devops IDC December 2014 Fortune 1000 Survey Situation Average cost of failure: >$500K Application development waste: 25% Solution 83% using or evaluating DevOps 21.4 % investing in Testing/QA tools Cautions Failure rate when using current tools: 80%
- 3. 3Spirent Communications www.spirent.com/solutions/devops SVM Build(s) SUT Commit Commands And Responses Source Code P4, Git, SVN, etc. Deliver Dev, CI, QA Labs Physical, virtual, hybrid environments Test and Lab Management Checkout Images Test Info Results data Logs Response Info Test I/P Test O/P Images Analytics Dashboards, ALM, PE Not Ready Ready CI/CD Orchestration Tools ContinuousIntegrationCI Continuous Test CT ContinuousDeliveryCD Continuous Change Management CCMPre-Flight Development Artifact Repository Images, tests, configs, logs, results Software Changes
- 5. 5Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Innovation Test strategy requires critical thinking which enables innovation and efficient CT which reduces wasted corrective work, frees time for innovation.
- 6. 6Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Quality CT is more than just quality measurement, it is an active part of quality development, verification and deployment.
- 7. 7Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Time-to-market Build CT for speed, and don’t stop!
- 8. 8Spirent Communications www.spirent.com/solutions/devops Continuous Testing means… Return-On-Investment Penny wise, pound foolish CT is a major factor affecting Return-On Investment
- 9. 9Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practices
- 10. 10Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #1 – Team and Culture Architect CT Skills Test Design Reviews Workflow Analysis Collaboration Training
- 11. 11Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #2 – CT-Ready Tools Restful APIs Cache and pipeline Virtual and physical Vertical aggregation Test topologies Program agnostic Large scale
- 12. 12Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #3 – Tools Integration Plug-in CI Tools CT Tools Plug-in CD Tools Plug-in CM Tools Restful APIs Restful APIs Documented Reusable DevOps ready Regression tests
- 13. 13Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #4 – Stability and Metrics Pre-Flight Static analysis Unit tests Integration Feature tests System tests Regression STOP REVERT ALLOW
- 14. 14Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #5 – CT Acceleration Powerful servers Test design Pre-load Pre-configure Scale horizontally No waiting! Thresholds Aggregate results
- 15. 15Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #6 – CT Analytics Test schedules Test selection Resources Test phases CT Controls CI, CD Controls Code reverts Promotions Releases
- 16. 16Spirent Communications www.spirent.com/solutions/devops Continuous Testing Best Practice #7 – Optimize CT Orchestration Orchestrating both physical and virtual test resources is important ! Physical Virtual Hybrid Catalogue Invoke Pipeline
- 17. 17Spirent Communications www.spirent.com/solutions/devops Continuous Testing Requires Expert Solutions Knowledge of CT best practices Cost of not getting it right Delays due to higher Ops priorities Ongoing cost of tool integrations, maintenance and enhancements 80% failure rate when using current tools IDC Fortune 1000 survey, December 2014
- 18. 18Spirent Communications www.spirent.com/solutions/devops Spirent CLEAR DevOps Solution Continuous Deployment (CD) Continuous Integration (CI) Plug-ins Plug-ins SUTTools Lab Management Physical, Virtual, Hybrid Lab Analytics ALM 6. Expertise and professional services 1. Test orchestration & lab management 2. Comprehensive suite of test tools 3. Physical, virtual and mixed hybrid labs 4. CI/CT/CD/CCM tools integration (EVCI) 5. CT analytics, ALM integration Orchestration Continuous Test (CT) CCM
- 19. 19Spirent Communications www.spirent.com/solutions/devops Implementation Strategies – Non-Disruptive Measured Progress Macro-Phases 1. Assessment to determine bottlenecks 2. Proof of concept 3. Horizontal integration 4. Vertical deployment Mini-Phases 1. Team & Culture 2. Tools integrations 3. Stabilize, measure 4. OptimizationsMacro- Phases Mini- Phases Initial Changing Optimized Micro-Phases 1. Change a little 2. Test 3. Deploy
- 20. 20Spirent Communications www.spirent.com/solutions/devops Real World Example – Network Equipment Manufacturer Metric Before After Major release (mo.) 6 3 Minor release (wks.) 4 2 # Features 113 150 Defects 1260 10 Integrations / day 0.5 100 Tests / day 0.3 10 Automated tests 5% 85% The primary contributing factor was CT !
- 21. 21Spirent Communications www.spirent.com/solutions/devops Summary The hidden secret of DevOps success is CT DevOps success will only be accomplished if CT is approached in accordance with best practices Don't fall into the “do-CT- yourself” trap!
Editor's Notes
- Spirent Communications is the premier global Test & Measurement company with HQ in UK. Spirent believes, like I do, that testing is critical to DevOps success and is committed to delivering solutions that address testing challenges for its customers. I discovered the importance of testing as a young engineer and stuck with it throughout my career. Kind of like the wise old advisor Erik in the famous DevOps book “The Phoenix Project” I may be considered a living “DevOps dinosaur” because I have been working on continuous integration, test, and deployment infrastructures for about 40 years since the mid-70’s. I published my first paper on the topic in 1977 when many of you in the audience were probably still in diapers. Even my college thesis project employed continuous test automation practices to deliver one of the worlds first operating microprocessor PBS systems at least a year years ahead of the first commercial PBX company that subsequently hired me. My first job involved developing automated test systems for one of the world’s first packet switch systems which became the world’s leading packet switch system partly due to the sophistication of the test capabilities we build for it and in it. Since then I have performed many roles as Senior Director for all the test technologies development for that global network manufacturer. I then have held roles such as VP of a major test equipment vendor, General Manager over a global test and measurement business, founder, CEO and CTO of the first network lab automation company, manager over a major DevOps organization and currently Senior Solutions Architect of Spirent’s Infrastructure Test Optimization Business Unit. In this talk I will lay bare for you some essential secrets to DevOps success. CT is often underestimated but is critical to business success. The talk explains how CT is essential to accomplish the four major goals of DevOps, describes seven best practices for CT that are important to ensure success, provides a warning about attempting to “do-CT-yourself”, a brief summary of Spirent’s CLEAR DevOps Solution and concludes with an offer for a free DevOps Assessment.
- This chart details some key findings of a recent IDC Fortune 1000 DevOps survey which is interesting because it demonstrates the importance of testing, QA tools and the poor track record of using current tools. With an average cost of failure at $500K and 25% of development efforts being wasted it is understandable why 83% of companies are using or evaluating DevOps. 21% of these companies are investing in Testing and QA tools which appears to be a wise choice because 80% report that their efforts failed when trying to use their current tools for DevOps. This is consistent with other recent market studies and my own experience and the thesis of this talk that DevOps is a powerful way to improve the performance of software delivery but to accomplish that requires investment into new DevOps-ready continuous test solutions . I reiterate -- 80% fail -- so true expertise is essential to get it right.
- *** Run this slide as animated *** This slide is a quick overview of DevOps tools infrastructures. DevOps has been described as tool-centric philosophy and in fact it is really four types of processes orchestrated by different classes of tools that are executed using physical, virtual and hybrid environments. The first process, called CI ,starts after the software completes a Pre-Flight test prior to integration to trunk. CI tools orchestrate the software changes, run static and unit tests, perform basic functional tests and produce build packages. The CT process used tools which orchestrate the test environment including the setup and configuration of the system-under-test and test and lab management function and run functional and system tests. The CD process used tools which manage deliverable package approvals, promotions and publish releases. The CM process uses tools run analytics used to determine code rework or promotion decisions. CI, CT, CD and CM tools work in concert to make DevOps work. It is interesting that although CI and CD are most talked about, CT and CM are at least as vital to DevOps success.
- DevOps has four major goals which are the same as the four major goals of the software development lifecycle (SDLC): namely to improve innovation, time-to-market, quality and ROI. So how do companies make a successful DevOps? What are they doing that’s different than the others? Well the hidden secret is not really a secret… but it’s something you perhaps haven’t focused on… DevOps success depends on CT. All the steps I discussed on the previous slide are important but *true* DevOps success will only be accomplished if CT is approached in accordance with best practices. This presentation is a story about continuous testing at the speed of DevOps. DevOps is all about Continuous Testing. Although much is published about the CI and CD , These are really the bookends of DevOps, it is CT that binds development to delivery. Without CT there is no CD, CI or CCM. Let’s look at how CT relates to the four goals of DevOps….
- The 1st goal of DevOps is to improve Innovation Continuous testing affects Innovation in two ways: First: Test strategy requires critical thinking to conceive testable products and the creative processes involving test strategies often result in innovative product design changes. Second: Efficient CT finds problems early, reduces wasted corrective effort that occurs if the problems are not found until later stages of development or deployment, which allows developers more time to spend on creative rather than corrective work. When not done effectively CT will actually impede innovation because the product may not satisfy unforeseen use cases and the backlog of test problems will overload the development team with corrective backlog work instead of creative work. Story: *** I recall a fun day when the innovative value of continuous testing was yelled into my face by a customer executive. A huge delegation from a major potential customer visited our R&D center to understand why our packet switch was able to operate with zero outages while several of our competitors that made up the infrastructure of their operation had crashed so often that the company was facing enormous government penalties and law-suits. When we explained how we build automated testing capabilities throughout our development, test, release and support processes but the tools are only available to customers that use our network equipment the head of the customer delegation stood up and literally screamed “How many network nodes do I need to buy to get these test tools!?” ***
- The 2nd goal of DevOps is to assure quality of the product or service. In my experience true quality is determined by the dependability of a product when used, rather than a simple measure of defects. CT is much more than a measurement tool that points out defects early. Because it is an integral part of all DevOps processes CT is an integral part of product development, deployment and even post-deployment maintenance. By integrating test processes deeply into DevOps CT ensures costly quality problems are avoided such as: Re-work Unsatisfied customers Non-compliance Bad Press Litigation Story: *** I recall being surprised myself when I was charged with a major quality study that clearly demonstrated a nearly exponential inverse relationship between the # of independent testers in an organization and the rate of customer found detects. It turned out that organizations with fewer independent testers were doing much more thorough testing using testing tools integrated directly into the development organization, and this continuous testing was driving quality so high it was getting more and more difficult to measure quality improvements. Exceptional quality due to integrated continuous testing was the primary reason the network system I was involved with became the dominant system throughout the world, and I became the youngest international industry quality representative in the history of the company.***
- Ok its late in the day and I figure you folks are tired and a picture of a cozy bunny is probably consistent with your mood right now. But surely you know the story of the tortoise and the hare. Well guess what the 3rd goal of DevOps is to reduce time-to-market. Done right, CT will accelerate finding important defects early in the development schedule Done wrong, CT will only rapidly accumulate failure reports that pile up and cause release delays even though many of the failures are not important. Doing CT right means building the entire test environment for Speed, so that problems can be handled in real time and you rarely have to stop the process which may have negative consequences to getting to the finish line on time, Story: *** The major packet switch vendor I worked for initially took 12 months to deliver each major release using protocol test suites that each took 2 weeks to run and analyze. We automated and scaled horizontally the protocol testing so test cycles were reduced to hours, deployment times were reduced to 6 weeks -- a nice 800% improvement. But we didn’t stop there. The bottleneck was then test analysis times. When the test analysis cycles were automated the deployment time reduced to 2 weeks – a whopping 2600% improvement, largely due to implementing continuous testing principles !***
- The forth and final goal of DevOps is to accomplish the first three goals without breaking the bank and to deliver a positive ROI. CT is a major factor affecting Return-On Investment. For example if the CT resource are not adequately shared and used efficiently then the costs of the resource soon spiral out of control. Story: *** The major switch vendor I worked for had 55 labs used by development, test and field trial testing organizations around the world. We determined that the equipment in these labs were utilized, on average, only 30%. By converting all the test systems, networking the lab resources into one orchestrated virtual lab, and triggering build testing directly from the build systems, the average utilization of the equipment increased to 75%, average wait times for developer testing reduced from weeks to minutes and simultaneously we accomplished global lab capital equipment savings of $90M per year ! *** It was at this juncture I leveraged my success as senior manager within a large networking company bases in the freezing climates of snowy Canada and moved my family to sun and sailboats and an executive role in a test and measurement company that is based in sunny southern California.
- So far we have been talking about ‘why’ CT is important to accomplishing the big four DevOps goals. So “what and how” differentiates CT practices that ensure success rather than failure? How can you “cross the chasm” from a situation where testing is a tactical cost of business to be a strategic process that directly affects business success? Lets walk through seven best practices that have proven to deliver the great results I have been talking about….
- The first best practice for CT is Team and Culture. Choose and manage your team strategically in accordance with the specific goals of CT. Assign an Architect that is responsible for CT overall. The architect must be an expert in CT practices and tools and set standards for details such as how long will the different CT cycles be? What level of test coverage is expected? What are the SLAs and workflows for resolving test failures? What are the actions and thresholds for test failures? The entire team needs CT skills training. This includes training for using test tools, automating test cases, test design, and test design reviews. CT requires collaboration between testing experts and code designers. This requires training and reinforcement because there is natural conflict between creating software and reporting failures of that software. Bring together both developers and testers for training to support building teamwork.
- The 2nd best practice for CT is CT -ready tools. Many existing tools are not CT-ready and can not accomplish best practices DevOps. This is a high level check-list of the most critical features that are needed in a CT-ready toolkit. Restful APIs are essential because CT tools must be able to integrate easily with everything else and each other. Features that allow all test activities to be cached and pipelined assure tests are not waiting around for some resource to be available. Test tools must be able to run virtualized for horizontal scaling yet optimized for execution on physical servers when time-critical critical performance testing is required. Results reporting should be separated from execution and aggregated vertically across as many test agents as needed to meet test cycle time SLAs. Test topologies and configurations need to be pre-defined and pipelined ahead of the tests. The test tools need to easy to program all types of tests in a program agnostic approach that is efficient to capture and create complex tests of all types. Finally the tools need to be able to scale from simple unit tests to very large scale systems tests that requires many test agents.
- The 3rd best practice for CT is Tools Integration. All CT tools must play seamlessly with CI, CD and CCM tools and other CT tools to ensure fast and reliable workflows. Tool integration is best accomplished with carefully designed plug-ins rather than custom ad-hoc scripts . The best Plug-ins are well documented and designed to be re-usable, to eliminate the need for each developer to create their own interface. The plugins themselves are subject to testing because each tool will be upgraded from time to time and the plugins need to be regression tested for each system change. Don’t forget to treat the plugins within the DevOps system also.
- The 4th best practice for CT is Stability and Metrics. CT is more than simply running a smoke test when a build is done. There are multiple levels of testing invoked as each software change progresses through the DevOps system. Each level needs to be measured and actions need to be taken for failed cases according to pre-determined workflows . Firstly CT system environment stability must be accomplished and constantly measured or the entire test environment will be a liability rather than a strategic asset. Pre-Flight testing metrics determine if the software does not meet its test pass rate and prevents the code from being integrated to trunk. Static analysis, unit tests and basic integration tests determine if the results of a build are good enough for more extensive integration testing, system testing and regression testing.. The metrics strategy sets thresholds for each test phase that determine whether failures are significant enough to revert the changes or it is necessary to stop the DevOps cycles long enough to fix the problems.
- The 5th and arguably the most important CT Best Practice is CT Acceleration. Since the underlying mechanism that drives success in DevOps is speed of the integration cycles, it should be clear that faster CT is preferred. Here is a check-list of practices that accelerate the tests, test schedules and test resources…. Use the most powerful computing resources you can afford and orchestrate them optimally for the types of tests they are performing Design the tests to focus on one verdict and get to the highest priority verdict quickly. Pre-load and pre-configure the next test while the current is running Run as many test agents in parallel as you can afford. Configure the entire system to remove bottlenecks and wait times. Set thresholds for failures and immediately release resources when triggered. Aggregate results on separate fast severs. One thing to keep in mind though….watch out for essential cases that can be compromised by continuous testing if not integrated properly. For example “ how to deal with long duration testing?, and how to integrate manual testing? into DevOps also need to be fully considered.
- The 6th Best Practice of CT is CT Analytics. CT analytics is more than just reporting pass/fail results. Best practices CT environments use statistical methods to control the CT process and also the CI and CD processes. As DevOps itself scales up, and accelerates the number of test results scales up rapidly to many thousands of results per cycle. If all of the results need to be processed manually the value of automated testing is wasted on laborious test analysis. So in best practices test analytics are not only descriptive they are predictive – they automatically keep track of test results trends correlated to software changes for each case and assign priorities, diagnostics and probable solutions rather than simply listing the pass/fail results. I refer you to some papers I wrote on the subject of automated change-based test selection and results analysis.
- The 7th best practice for CT Orchestration. In my experience only the most advanced CT environments are orchestrating their test topologies but the optimizations afforded by this is outstanding so amazes me that more are not doing this. Perhaps they are not aware of tools and workflows that implement this practice? Do you know the difference between automation and orchestration? I am referring to the automated management of test topologies and configuration data system under test and test tools as required for each test case Best practices test orchestration tools support physical, virtual and hybrid test resources and their interconnections equally. This includes the ability to inventory, catalogue, invoke, pre-load, implement and re-configure systems and networked test topologies in concert with pipe-lined test schedules.
- The picture says it all. The “handy” dapper business man on the tree obviously decided he can save some time and money by doing this “simple task” himself on his lunch break rather than hiring a professional tree trimmer. He appears to have the right tool and the personal know-how to operate the tool to accomplish an intended result of cutting the unwanted tree limb, but he doesn’t seem to fully understand all of the best practices so he is about to experience a serious unintended consequence. My advise is to be wary when your team tells you they “don’t need help with CT” I am sure they are smart but before you decide to “do-CT-yourself ask yourself: Does your team have intimate knowledge of CT best practices? Can you afford the cost of not getting it right the first time? Can your organization afford the delays that inevitably occur when they get interrupted by higher priority product escalations. Can you afford the ongoing cost of tools integrations with new tools and maintenance and enhancements that will surely occur in the future? Remember those IDC Fortune 1000 report…..last line on chart … 80% fail ! My advice is think more than twice before you decide to “do-CT-yourself”!
- ** Animated slide ** Spirent’s CLEAR DevOps Solution is a comprehensive toolkit that embodies all of the best practices and offers professional services to get it integrated into any customer tool chain. At the heart of the solution is a test orchestration system that has all of the features needed to orchestrate not only topologies and configurations of the system-under-test and any vendor test tools suitable for both small and large scale DevOps. Spirent offers wide range of powerful test tools for conformance, performance, network, security, feature and security testing that play well with Spirent orchestration tools. The entire suite of test tools can be operated in physical, virtualized or hybrid lab environments as needed to scale tests as horizontally as needed for faster test cycles and as fast as possible to overdrive the world’s faster performance tests. Through EVCI plugins with Jenkins for instance the tools can be easily integrated with any CI, CD and CM tools that can also play with CI orchestration toolkits. The solution provides custom dashboards and integrates with ALM systems for advanced analytics as needed for CT to not get bogged down at the analysis phase. Finally the solution includes proven expertise and professional services scalable as needed to ensure the entire system is assessed, implemented according to best practices and seamlessly integrated into existing customer environments without disrupting existing releases.
- A word about implementation and the need for a deep understanding of professional services that assure best practices implementation without disruption. Poorly planned implementation usually results in poor outcomes no matter how good an idea is. All the parts need to be implemented in concert. Implementation strategies that minimally disrupt the current environment and provide measured progress are usually best. A wise implementer is aware that each change needs to fit like clockwork within a macro and micro environment or else the entire system may become broken. A multi-level phased approach is recommended similar to the different interworking layers of DevOps continuous improvement concepts. At the Macro-Phase level start with best practices GAP assessments, design execute proof-of-concept experiments that target the highest improvement areas, then integrate the chosen solution components horizontally with existing practices, and finally vertically deploy the solution components into the existing operations to minimize disruptions. At the Mini-phase level take care of team, culture, tool, environment, measures and then optimize each Macro phase component in order of priority. . At the micro-phase level use Agile and DevOps principles for each change: Change-a-little, test it, deploy it and loop.
- ** Animation ** This chart is a clear example how Results are the point. One of our customers, a major network manufacturer was having problems with feature velocity, release delays, quality, time-to-market, excessive lab costs. Major releases were taking 6 months or more. Features were getting delayed beyond customer commitment dates, the # of customer reported defects were alarmingly high and rising, and internal integration and test processes were bogged down despite rapidly rising costs. In the first year after implementing Spirent’s CLEAR DevOps Solution the time-to-market for major and minor release were reduced by 50%, feature velocity was increased 30%, customer reported defects were dramatically reduced, internal integration and test cycles were improved up to 100 times. As you can see fro the graph on the right, the customer made a sizable strategic investment and it is paying off. The ROI is now on the rise dramatically with net $8Million returned within two years after the DevOps system investment and deployment. The last three lines of the chart is a good indication how these dramatic improvements were accomplished. Automated testing with CT best practices enabled faster cycles, more test automation integrated with development and much faster incremental feedback loops at the micro, mini and macro levels.
- Ok so let me recap the thesis of my talk which hopefully has been made clear by now. While it should not be a secret, for some reason many organizations, even many scholarly publications, have not fully recognized the strategic nature of getting Continuous Testing right in order for DevOps to accomplish it’s primary success goals of innovation, quality, time to market, and ROI. There is much more to CT than just running some tests after a build. A deep and broad understanding of CT best practices is vital to putting together a winning DevOps infrastructure. So we recommend that you don’t fall out of the tree by trying to “do-CT-yourself”. Indeed it can be penny wise and pound foolish not to engage with experienced professionals.
- I conclude my talk with a call to action for you to thoroughly understand DevOps best practices, and the GAP between your current practices and best practices. Spirent has a DevOps Best Practices Assessment tool that can help you determine your current practices level and GAP. It is a straight forward spreadsheet your team can fill out themselves or we would be happy to walk you through it. We offer a DevOps best practices assessment service free of charge. If interested contact myself, stop by our table outside, or contact your local Spirent sales. We also offer free of charge DevOps Solutions Blueprint whitepaper at www.spirent.com/solutions/devops or stop by the booth for a free copy. Thank-you for your attention and I wish you the best in your DevOps efforts!