Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Moving to Continuous
DeliveryWithout Breaking
Your Code
Andrew Phillips, 18 June 2015
2 Copyright 2014. Confidential – Distribution prohibited without permission
Agenda
▪ The Need For Speed
▪ The Two Faces of...
3 Copyright 2014. Confidential – Distribution prohibited without permission
About Me
▪ VP Products for XebiaLabs
▪ Lots of...
4 Copyright 2014. Confidential – Distribution prohibited without permission
About XebiaLabs
We build tools to solve proble...
5 Copyright 2014. Confidential – Distribution prohibited without permission
The Need For Speed
▪ Every business is an IT b...
6 Copyright 2014. Confidential – Distribution prohibited without permission
The Need For Speed
▪ What is so compelling abo...
7 Copyright 2014. Confidential – Distribution prohibited without permission
TheTwo Faces of CD
▪ A lot of focus right now ...
8 Copyright 2014. Confidential – Distribution prohibited without permission
TheTwo Faces of CD
▪ CD = Execution + Analysis
9 Copyright 2014. Confidential – Distribution prohibited without permission
TheTwo Faces of CD
▪ CD = Execution + Analysis...
10 Copyright 2014. Confidential – Distribution prohibited without permission
TheTwo Faces of CD
▪ CD = Execution + Analysi...
11 Copyright 2014. Confidential – Distribution prohibited without permission
SPECIFY INTEGRATE RELEASEREGRESSION
VALUE CHA...
12 Copyright 2014. Confidential – Distribution prohibited without permission
SPECIFY INTEGRATE RELEASEREGRESSION
VALUE CHA...
13 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Challenges
▪ Many test t...
14 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Challenges
▪ Thousands o...
15 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Best Practices
▪ Focus o...
16 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Best Practices
▪ “Slice ...
17 Copyright 2014. Confidential – Distribution prohibited without permission
Making Sense ofTest Results
▪ Real go/no go d...
18 Copyright 2014. Confidential – Distribution prohibited without permission
Example Job Distribution
Build Deploy Int. Te...
19 Copyright 2014. Confidential – Distribution prohibited without permission
Example Job Distribution
Build Deploy Int. Te...
20 Copyright 2014. Confidential – Distribution prohibited without permission
Making Sense ofTest Results
Executing tests f...
21 Copyright 2014. Confidential – Distribution prohibited without permission
Making Sense ofTest Results
▪ Ultimate analys...
22 Copyright 2014. Confidential – Distribution prohibited without permission
A Central Hub for Application Quality
What is...
23 Copyright 2014. Confidential – Distribution prohibited without permission
BeyondTest Automation
Can we go further? How ...
24 Copyright 2014. Confidential – Distribution prohibited without permission
BeyondTest Automation
It’s a bit of an open q...
25 Copyright 2014. Confidential – Distribution prohibited without permission
BeyondTest Automation
Middle ground:
▪ Label ...
Thank you!
Upcoming SlideShare
Loading in …5
×

Moving to Continuous Delivery Without Breaking Your Code

280 views

Published on

Slides from the "" presentation at CD Summit East 2015:

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Moving to Continuous Delivery Without Breaking Your Code

  1. 1. Moving to Continuous DeliveryWithout Breaking Your Code Andrew Phillips, 18 June 2015
  2. 2. 2 Copyright 2014. Confidential – Distribution prohibited without permission Agenda ▪ The Need For Speed ▪ The Two Faces of CD ▪ Testing is Changing ▪ A Central Hub for Application Quality For Your Pipeline ▪ Beyond Test Automation: Active Test Optimization
  3. 3. 3 Copyright 2014. Confidential – Distribution prohibited without permission About Me ▪ VP Products for XebiaLabs ▪ Lots of enterprise software development on high-performance systems ▪ Been on both sides of the “Dev…Ops” fence ▪ Active open source contributor and committer: Apache jclouds and others ▪ Microservices, reactive & Scala fan ▪ Regular meetup, conference etc. presenter
  4. 4. 4 Copyright 2014. Confidential – Distribution prohibited without permission About XebiaLabs We build tools to solve problems around DevOps and Continuous Delivery at scale
  5. 5. 5 Copyright 2014. Confidential – Distribution prohibited without permission The Need For Speed ▪ Every business is an IT business − Known as “Software-defined Enterprise”, even traditionally brick and mortar business is becoming software based ▪ Customers demand that you deliver new features faster whilst maintaining high levels of quality ▪ If you don’t, your competitor probably will
  6. 6. 6 Copyright 2014. Confidential – Distribution prohibited without permission The Need For Speed ▪ What is so compelling about CD? ▪ Business initiative with cool technical implementation ▪ “CD eats DevOps for breakfast as the business eats IT”
  7. 7. 7 Copyright 2014. Confidential – Distribution prohibited without permission TheTwo Faces of CD ▪ A lot of focus right now is on pipeline execution ▪ …but there’s no point delivering at light speed if everything starts breaking ▪ Testing (= quality/risk) needs to be a first-class citizen of your CD initiative!
  8. 8. 8 Copyright 2014. Confidential – Distribution prohibited without permission TheTwo Faces of CD ▪ CD = Execution + Analysis
  9. 9. 9 Copyright 2014. Confidential – Distribution prohibited without permission TheTwo Faces of CD ▪ CD = Execution + Analysis ▪ = Speed + Quality
  10. 10. 10 Copyright 2014. Confidential – Distribution prohibited without permission TheTwo Faces of CD ▪ CD = Execution + Analysis ▪ = Speed + Quality ▪ = Pipeline orchestration + ..?
  11. 11. 11 Copyright 2014. Confidential – Distribution prohibited without permission SPECIFY INTEGRATE RELEASEREGRESSION VALUE CHAINCONCEPT CASH TestEffort USER ACCEPTANCEDESIGN BUILD TEST Testing is Changing
  12. 12. 12 Copyright 2014. Confidential – Distribution prohibited without permission SPECIFY INTEGRATE RELEASEREGRESSION VALUE CHAINCONCEPT CASH TestEffort USER ACCEPTANCE USER ACCEPTANCE Acceptance Driven Testing “I add value by sharpening the acceptance criteria of requested features” Automate ALL “Test automation serves as the safety net for my new functionality: I focus on running the appropriate tests continuously during the iterations” T E S T D E S I G N B U I L D D B T T D Development = Test Test = Development “Testing is transforming to a automation mindset and skill instead of a separate activity” Testing is Changing
  13. 13. 13 Copyright 2014. Confidential – Distribution prohibited without permission Testing is Changing: Challenges ▪ Many test tools for each of the test levels, but no single place to answer “Good enough to go live?” ▪ Requirements coverage is not available − “Did we test enough?” ▪ Minimize the mean time to repair − Support for failure analysis JUnit, FitNesse, JMeter, YSlow, Vanity Check, WireShark, SOAP-UI, Jasmine, Karma, Speedtrace, Selenium, WebScarab, TTA, DynaTrace, HP Diagnostics, ALM stack AppDynamics, Code Tester for Oracle, Arachnid, Fortify, Sonar, …
  14. 14. 14 Copyright 2014. Confidential – Distribution prohibited without permission Testing is Changing: Challenges ▪ Thousands of tests makes test sets hard to manage: − “Where is my subset?” − “What tests add most value, what tests are superfluous?” − “When to run what tests?” ▪ Running all tests all the time takes too long, feedback is too late ▪ Quality control of the tests themselves and maintenance of testware ▪ Tooling overstretch
  15. 15. 15 Copyright 2014. Confidential – Distribution prohibited without permission Testing is Changing: Best Practices ▪ Focus on functional coverage, not technical coverage ▪ Say 40 user stories, 400 tests − Do I have relatively more tests for the more important user stories? − How do I link tests to user stories/features/fixes? ▪ Metrics − Number of tests − Number of tests that have not passed in <time> − Flaky tests − Duration
  16. 16. 16 Copyright 2014. Confidential – Distribution prohibited without permission Testing is Changing: Best Practices ▪ “Slice and dice” your test code − Responsible team − Topic − Functional area − Flaky − Known issue − etc. ▪ Radical parallelization − Fail faster!
  17. 17. 17 Copyright 2014. Confidential – Distribution prohibited without permission Making Sense ofTest Results ▪ Real go/no go decisions are non-trivial − No failing tests − 5 % of failing tests − No regression (tests that currently fail but passed previously) − List of tests-that-should-not-fail ▪ Need historical context ▪ One integrated view ▪ Data to guide improvement
  18. 18. 18 Copyright 2014. Confidential – Distribution prohibited without permission Example Job Distribution Build Deploy Int. Tests Test Test Test Perf. Tests Build Deploy Int. Tests Test Test Test Perf. Tests
  19. 19. 19 Copyright 2014. Confidential – Distribution prohibited without permission Example Job Distribution Build Deploy Int. Tests Test Test Test Perf. Tests Build Deploy Int. Tests Test Test Test Perf. Tests Simple pipelines – scattered test results
  20. 20. 20 Copyright 2014. Confidential – Distribution prohibited without permission Making Sense ofTest Results Executing tests from Jenkins is great, but… ▪ Different testing jobs use different plugins or scripts, each with different visualization styles ▪ No consolidated historic view available across jobs ▪ Pass/Unstable/Fail is too coarse − How to do “Passed, but with known failures”?
  21. 21. 21 Copyright 2014. Confidential – Distribution prohibited without permission Making Sense ofTest Results ▪ Ultimate analysis question (“are we good to go live?”) is difficult to answer ▪ No obvious solution for now, unless all your tests are running through one service
  22. 22. 22 Copyright 2014. Confidential – Distribution prohibited without permission A Central Hub for Application Quality What is needed: 1. A single, integrated overview of all the test (= quality, risk) information related to your current release 2. …irrespective of where or by whom the information was produced 3. The ability to analyze and “slice and dice” the test results for different audiences and use cases 4. The ability to access historical context and other test attributes to make real- world “go/no-go” decisions
  23. 23. 23 Copyright 2014. Confidential – Distribution prohibited without permission BeyondTest Automation Can we go further? How about 5. The ability to use the aggregated test results, historical contexts and other attributes to invoke tests more intelligently?
  24. 24. 24 Copyright 2014. Confidential – Distribution prohibited without permission BeyondTest Automation It’s a bit of an open question: ▪ Google: it’s too expensive and time-consuming to run all the tests all the time - automated selection of a subset of tests to run ▪ Dave Farley: if you can’t run all the tests all the time, you need to optimize your tests or you have the wrong tests in the first place
  25. 25. 25 Copyright 2014. Confidential – Distribution prohibited without permission BeyondTest Automation Middle ground: ▪ Label your tests along all relevant dimensions to ensure that you can easily select a relevant subset of your tests if needed ▪ Consider automatically annotating tests related to features (e.g. added/modified in the same commit), or introducing that as a practice ▪ Use data from your test aggregation tool to ignore flaky/”known failure” tests (and then fix those flaky tests, of course ;-))
  26. 26. Thank you!

×