• Like
  • Save

Why Testing Needs to be #1 for Continuous Delivery

  • 180 views
Uploaded on

Slides from the "Why Testing Should be Number One for your Continuous Delivery initiative" webinar on July 8th, 2014

Slides from the "Why Testing Should be Number One for your Continuous Delivery initiative" webinar on July 8th, 2014

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
180
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Tell story from CIO of a big bank: “we need to deliver faster or we will go out of business”
  • Typical tools used include FitNesse, Cucumber, JBehave for functional tests, JMeter, LoadRunner, PerformanceCenter for performance related tests, Fortify / Sonar
  • Google story
  • Do you mean ‘regression’ here? Interesting. Most organizations do regression via nightlies (anti-pattern) and new functionality earlier in the pipeline (at each commit). So, in some cases, check the regression first before you test the added functionality. It depends. So, it should be flexible!
  • Test context: relevant logging, infra performance, etc.
  • Focus on tests that add value.
    Yet, focus on unit testing (highest performance, automated by default, and lowest level, so more path coverage possible).
  • @Heather: please align/update

Transcript

  • 1. WhyTesting Needs to be #1 for Continuous Delivery Andrew Phillips, VP Products | 08 July 2014
  • 2. 2 Copyright 2014. About Me ▪ VP Products for XebiaLabs ▪ Lots of enterprise software development on high-performance systems ▪ Been on both sides of the “Dev…Ops” fence ▪ Active open source contributor and committer: jclouds, Akka, Gradle and others ▪ Cloud, PaaS & Scala fan ▪ Regular meetup, conference etc. presenter
  • 3. 4 Copyright 2014. Housekeeping ▪ This webinar is being recorded ▪ Links to the slides and the recording will be made available after the presentation ▪ You can post questions via the GoToWebinar Control Panel
  • 4. 5 Copyright 2014. Agenda ▪ We’re All Going Continuous Delivery ▪ Why Testing Needs To Be #1 ▪ Testing Challenges: From Not Enough to Too Many ▪ Managing Tests in a CD Organization ▪ What Do I Need To Do Now? ▪ Q & A
  • 5. 6 Copyright 2014. We’re All Going Continuous Delivery “Continuous delivery is a set of patterns and best practices that can help software teams dramatically improve the pace and quality of their software delivery.”
  • 6. 7 Copyright 2014. We’re All Going Continuous Delivery ▪ Faster time-to-market ▪ Increased application quality ▪ Greater customer responsiveness
  • 7. 8 Copyright 2014. We’re All Going Continuous Delivery ▪ Faster time-to-market ▪ Increased application quality ▪ Greater customer responsiveness ▪ …who would say no to that?
  • 8. 9 Copyright 2014. We’re All Going Continuous Delivery ▪ Faster time-to-market ▪ Increased application quality ▪ Greater customer responsiveness ▪ …who would say no to that? ▪ Hot trend currently getting a lot of attention and becoming a C-level topic as part of the DevOps transformation and due to competitive pressure ▪ Your only security is in being the disruptor ▪ Disruption demands delivering fast & frequently, with high quality
  • 9. 10 Copyright 2014. WhyTesting NeedsTo Be #1 ▪ “…dramatically improve the pace and quality of their software delivery”
  • 10. 11 Copyright 2014. WhyTesting NeedsTo Be #1 ▪ “…dramatically improve the pace and quality of their software delivery” ▪ Many initiatives we currently see are looking to build delivery pipelines, automate environment creation and app deployment etc. etc. ▪ That’s all very well, but the most important goal of a pipeline is to allow you to make an informed (and, at some stage, perhaps automated) Go/No-go decision!
  • 11. 12 Copyright 2014. WhyTesting NeedsTo Be #1 ▪ “…dramatically improve the pace and quality of their software delivery” ▪ What do you need to do that? Tests, tests and more tests! ▪ Few things will dampen the executive sponsors’ enthusiasm for CD more quickly than a couple of embarrassing post-release outages
  • 12. 13 Copyright 2014. WhyTesting NeedsTo Be #1 ▪ Streamlining development: Accurate tests will help you determine when you’ve done just enough to implement new features ▪ Managing risk: Sufficient testing will help ensure your CD initiative doesn’t get stopped before it’s even got off the ground ▪ Increasing quality: Further testing will allow you to determine whether the new deliverable is better than what’s currently running
  • 13. 14 Copyright 2014. WhyTesting NeedsTo Be #1 Testing should be the most important element of your pipeline – the rest is just plumbing
  • 14. 15 Copyright 2014. Testing Challenges: From Not Enough toToo Many ▪ Status quo in many situations: no automated tests and only limited manual tests ▪ Will severely limit the throughput capacity and predictability of your delivery pipeline
  • 15. 16 Copyright 2014. Testing Challenges: From Not Enough toToo Many ▪ Status quo in many situations: no automated tests and only limited manual tests ▪ Will severely limit the throughput capacity and predictability of your delivery pipeline ▪ Other extreme: lots and lots of tests using many different tools ▪ How to aggregate this information to allow you to make a confident Go/No-go decision?
  • 16. 17 Copyright 2014. Testing Challenges: From Not Enough toToo Many ▪ Problem today: Test failures are hard to reproduce. Not a new issue, but gets much worse with more tests and the need for faster throughput ▪ Future problem: So many tests that it simply takes too long and costs too much money to run them ▪ Also introduces a signal-to-noise problem: how much does one failure stand out in a crowd of thousands of less meaningful tests that always pass?
  • 17. 18 Copyright 2014. Testing Challenges: From Not Enough toToo Many ▪ Problem today: Test failures are hard to reproduce. Not a new issue, but gets much worse with more tests and the need for faster throughput ▪ Future problem: So many tests that it simply takes too long and costs too much money to run them ▪ Also introduces a signal-to-noise problem: how much does one failure stand out in a crowd of thousands of less meaningful tests that always pass? ▪ How to do the right amount of testing for the current context and the desired risk/cost/speed tradeoff??
  • 18. 19 Copyright 2014. ManagingTests in a CD Organization 1. Ensure you have tests to measure all relevant aspects of risk and quality for your application − Probably will need multiple tools to achieve this
  • 19. 20 Copyright 2014. ManagingTests in a CD Organization 1. Ensure you have tests to measure all relevant aspects of risk and quality for your application − Probably will need multiple tools to achieve this 2. Ensure the first set of tests verify the most critical aspects of the system − May not even be the new functionality! − Most important to make sure key use cases still work when a new release goes out
  • 20. 21 Copyright 2014. ManagingTests in a CD Organization 1. Ensure you have tests to measure all relevant aspects of risk and quality for your application − Probably will need multiple tools to achieve this 2. Ensure the first set of tests verify the most critical aspects of the system − May not even be the new functionality! − Most important to make sure key use cases still work when a new release goes out 3. Capture as much as you need when you run your tests − If you have enough information about what was going on in the app to determine what the problem is, you will spend less time trying to reproduce test failures
  • 21. 22 Copyright 2014. ManagingTests in a CD Organization 4. Build out test coverage based on the decreased risk/increased quality information available − Not necessarily by “test category”!
  • 22. 23 Copyright 2014. ManagingTests in a CD Organization 4. Build out test coverage based on the decreased risk/increased quality information available − Not necessarily by “test category”! 5. More tests are not always better! − Tradeoff here between added information and runtime cost, maintenance cost, complexity of the overall test picture etc.
  • 23. 24 Copyright 2014. ManagingTests in a CD Organization 4. Build out test coverage based on the decreased risk/increased quality information available − Not necessarily by “test category”! 5. More tests are not always better! − Tradeoff here between added information and runtime cost, maintenance cost, complexity of the overall test picture etc. 6. Choose tests according to the context − E.g. run more tests in the areas of the system that are being changed in the current pipeline run
  • 24. 25 Copyright 2014. ManagingTests in a CD Organization 4. Build out test coverage based on the decreased risk/increased quality information available − Not necessarily by “test category”! 5. More tests are not always better! − Tradeoff here between added information and runtime cost, maintenance cost, complexity of the overall test picture etc. 6. Choose tests according to the context − E.g. run more tests in the areas of the system that are being changed in the current pipeline run 7. Actively manage your test set − Looking to minimize the signal-to-noise ratio of the test output − A tests that never fails can be as wasteful as a test that never succeeds!
  • 25. 26 Copyright 2014. What Do I NeedTo Do Now? 1. Identify a baseline of which tests currently exist and how long they take to run − May be compatible with the targets you have set for your delivery pipeline!
  • 26. 27 Copyright 2014. What Do I NeedTo Do Now? 1. Identify a baseline of which tests currently exist and how long they take to run − May be compatible with the targets you have set for your delivery pipeline! 2. Determine the most critical use cases for your application and ask: − “What is the likelihood of breaking this use case?” − “How bad is it if this use case no longer works?” − “How quickly can we fix it if it breaks?” − “Do we have tests to cover this use case? Manual or automated? How many of them do I need to run to be confident enough that the use case still works well (enough)?” − …
  • 27. 28 Copyright 2014. What Do I NeedTo Do Now? 3. If you are missing tests, add time and budget to create these into your pipeline initiative − You may also decide to invest in better mechanisms to fix things quickly, for example! − Determine which tests to create based on use cases rather than test categories − Also ensure you capture information about the systems under test as the test runs
  • 28. 29 Copyright 2014. What Do I NeedTo Do Now? 3. If you are missing tests, add time and budget to create these into your pipeline initiative − You may also decide to invest in better mechanisms to fix things quickly, for example! − Determine which tests to create based on use cases rather than test categories − Also ensure you capture information about the systems under test as the test runs 4. Ensure your tests are linked to the use cases and parts of the system they cover − Without this, choosing the most appropriate test set for the context is hard!
  • 29. 30 Copyright 2014. What Do I NeedTo Do Now? 3. If you are missing tests, add time and budget to create these into your pipeline initiative − You may also decide to invest in better mechanisms to fix things quickly, for example! − Determine which tests to create based on use cases rather than test categories − Also ensure you capture information about the systems under test as the test runs 4. Ensure your tests are linked to the use cases and parts of the system they cover − Without this, choosing the most appropriate test set for the context is hard! 5. Start thinking about how to keep your test set “relevant”
  • 30. 31 Copyright 2014. Next Steps ▪ Get started today! go.xebialabs.com/XL-Platform-Trial.html ▪ Learn more about XebiaLabs: www.xebialabs.com/products/ ▪ Stay informed: blog.xebialabs.com @XebiaLabs vimeo.com/xebialabs
  • 31. 32 Copyright 2014. Q & A ▪ Over to you! ▪ Input your questions in the control panel now.
  • 32. Thank You!