Continuous Delivery at


Published on

Continuous delivery is a practice that enables teams to release code at any time, based on changing business requirements. However, continuous delivery requires a substantial investment in infrastructure and possibly fundamental architectural changes to support the process. Anti-patterns that would render a continuous delivery pipeline a burden rather than a beneficial tool for continuous delivery must be avoided. Seng Lin Shee describes how has transitioned from small scale web deployments to rapidly deploying new features without compromising product quality—all while not adding more complicated processes in the development lifecycle. A disciplined team is necessary to successfully use a delivery pipeline to its utmost potential. Seng Lin shares his team’s experience in the development of pipeline testing processes, introducing different sets of environments for different testing purposes, isolation testing, and test case management in a continuous delivery environment. Take away key ideas for implementing continuous delivery in your organization.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Continuous Delivery at

  1. 1.     BW11 Session  6/5/2013 3:45 PM                "Continuous Delivery at"       Presented by: Seng Shee                 Brought to you by:        340 Corporate Way, Suite 300, Orange Park, FL 32073  888‐268‐8770 ∙ 904‐278‐0524 ∙ ∙
  2. 2. Seng Lin Shee Leading the testing effort for the API team of, Seng Lin Shee educates the team and defines test strategies and direction of the testing effort. He has introduced the team to behavior-driven design and test-driven design methodologies, and contributed to the design of the continuous delivery pipeline within the company. Seng Lin’s experience ranges from software development and testing, to router/switch manufacturing, to microprocessor architecture research. Seng Lin was formerly a software development engineer in test at Microsoft where he worked on Windows Server 2008 R2, Active Directory Federation Services, and Access Control Services.  
  3. 3. Continuous Delivery at Seng Lin Shee @ y Take Away from this Presentation • Understanding of Continuous Delivery • Role of Continuous Delivery in web services •I Impact on the teams at h A • Strategies for an effective Continuous Delivery pipeline • Changes in responsibilities and expectations 2
  4. 4. Agenda 1. What is Continuous Delivery 2. Continuous Delivery Architecture 3. Software Lifecycle via Continuous Delivery 3 S f Lif l i C i D li 4. Testing Methodology 5. Roles and Responsibilities (Mindsets) 6. Learnings / Pitfalls 3 The Issue… • Traditional deployment strategy of web services: – Involvement of entire engineering team – Manual monitoring – Manual test execution – Tests executed within unstable environment 4
  5. 5. Value Stream • Flow of materials and information required to bring a  product or service to a consumer p • Value stream map: value add and wait time – Product concept – Product discovery – Product planning – Development – Testing – Release • We need to be more Agile! 5 Philosophy • “The essence of my philosophy to software delivery is to  build software so that it is always in a state where it  could be put into production. We call this Continuous  Delivery because we are continuously running because we are continuously running  a deployment pipeline that tests if this software is in a  state to be delivered.” (Martin Fowler) 6
  6. 6. What is Continuous Delivery? Continuous Delivery Continuous Deployment Continuous Integration 7 Cycle Time • How to optimize cycle time – Teach developers how to write production‐ready code – Running Continuous Integration and tests on production‐like  systems – Implement an Agile development process  – Instituting cross‐functional teams – Automating the build, test and release process – Implement Continuous Delivery using a Deployment Pipeline 8
  7. 7. Learning Release Management • Continuous Delivery is about keeping your software in a  state where it could be released into production at any  time AND being able to actually release it when the  business wants to. business wants to. 9 Release Management Principles and Practices • Release small changes frequently rather than large  changes infrequently p y • Automated Deployment • Automated Configuration Management – OS, services, applications, everything – No more manual configuration • Automated processes are self documenting • Automated change tracking to production resources • Automated deployment smoke tests (test the deploy) • Automation, automation, automation… 10
  8. 8. Tooling • Provides the following services – Distributed build system Distributed build system – Continuous integration – Workflow engine (deployment pipeline) – Job control system – Dashboard (web application) • Hub spoke design multi platform Hub‐spoke design, multi‐platform – Single Go server – Multiple Go agents (one or more per machine) 11 Continuous Delivery Architecture
  9. 9. Traditional Deployment Environments DEV Environment STAGE Environment LIVE Environment Version Control System 13 Towards Continuous Delivery Build, Unit Tests & Packaging Tests critical functions Build 14 Tests remaining functionalities Acceptance Preproduction Tests all functionalities LIVE
  10. 10. Then & Now THEN NOW • DEV & STAGE environments are easily broken • Weekly rolls take hours to complete testing • All environments are provided with isolated boxes for testing • Weekly rolls take no more than 15 minutes 15 Maturity of Pipeline 16 Continuous Delivery pipeline rolls with tight supervision Automatic roll during business hours No deployment issues d after 1 month Continuous Delivery pipeline rolls binaries every fortnight Smok tests and more ke robust tests Rolls are performed in a concerted effort every fortnight Basic tests are in p placed in pipeline No pipeline MATURITY LEVEL
  11. 11. Software Lifecycle via Continuous Delivery Kanban Principals • Very much suited for our release cycles • Favored by Project Managers •A A story is DONE when a feature is DEPLOYED to a  i DONE h f i DEPLOYED particular ENVIRONMENT BACKLOG 18 IN DEVELOPMENT DONE
  12. 12. Testing Methodology Smoke Tests during Deployment Internal smoke tests Dependency smoke tests 20
  13. 13. Profile of Tests at each Quality Gate • Acceptance Tests Tests critical functions • Regression Tests •P f Performance Tests T • Tests are categorized into – READY – QUARANTINE – REGRESSION Acceptance 21 GO Pipeline FAQ • READY These are tests that are in the Acceptance Test project which  are ready, but with stories not completed. These tests will not  block the pipeline. We should be observing this group of test block the pipeline We should be observing this group of test • QUARANTINE These are tests which are known to fail, due to a certain bug  (which will take time to fix). We do not mark them as ignored  because these are still valid tests. • REGRESSION These are normal test that should pass 100% of the time. If they  fail, please retry the respective block of test. 22
  14. 14. Tests Parallelism • Pipelines should be made as wide and as short as  possible • GOAL – Reduce time for testing • CHALLENGES – Tests should  not interfere with one another – Tests should not depend on sequence of execution – Non‐performance tests should not overload server Non‐performance tests should not overload server • SOLUTION – Parallelism via namespaces / test categories / test classes 23 Roles and Responsibilities (Mindset)
  15. 15. Continuous Integration Key Principles and Practices • Maintain a Single Source Repository • Everyone Commits To the Mainline Every Day •A Automate a FAST, self‐testing build (of the Mainline  FAST lf i b ild ( f h M i li check‐ins) on an integration machine • Test in a Clone of the Production Environment • Make it Easy for Anyone to Get the Latest Executable •E Everyone can see what's happening h ' h i 25 Continuous Delivery Key Principles • Single source code branch (e.g. trunk/mainline) • Single build (no DEV and PROD build definition) •D l Deploy the same way to every environment h i • Configuration management • Infrastructure as code Execution time # of tests of tests Pipeline progress 26
  16. 16. Continuous Delivery Values and Practices • Automated testing and deployment • Workflow handles check‐in to release (deployment  pp pipeline) ) – Pipeline execution instances are versioned – Pipeline output becomes an artifact in repository • Teams own their build and workflow – Use Ant/NAnt script to define build  – Use Continuous Integration tool to design workflow Use Continuous Integration tool to design workflow – Elements: pipeline group, pipeline, stage, job and task 27 Continuous Delivery Anti‐Patterns • Multiple branches • Feature branches that are long‐lived (> 1 day) •C Commit tests taking longer than 10 minutes i ki l h 10 i • Team culture allows ignoring what happens after check‐in • Multiple builds in a pipeline 28
  17. 17. Developers Own the Releases • Developers are testers • Responsible to address checked‐in builds that fail in any  pp pipeline stages g • File bugs, mark tests are quarantined and move on with  development 29 Broken Pipelines happen…stuff happens… • Run local builds with unit tests first if your check‐ins are  constantly causing build breaks (pipeline breaks) p • Breaks are transparent; fix within 10 minutes, or roll back • “Always in a state where it could be put into production” 30
  18. 18. Learnings and Pitfalls Warm up time of services • A .NET Web Service (IIS hosted services) cannot handle  requests immediately  upon deployment. g p g • Loading and compiling of .NET assemblies on first  request. • The DEPLOY job has been configured to performed  smoke tests which exercises critical areas of code, a  cluster at a time. Product code has to be engineered for: • Product code has to be engineered for: – Quick boot up time – Robust against frequent recycling of application pools 32
  19. 19. Extreme Long Release Latency • A deployment pipeline has the following goal – Increase throughput via simultaneous execution of stages – Early fault detection • Longer delay between CheckIn’s and Releases due to – Increase in the number of quality gates – Repetition of the exact tests in differing quality gates – Longer sequential tests • This impacts the Time To Resolve when a quick fix is This impacts the Time‐To‐Resolve when a quick fix is  needed 33 Backward and Forward Compatibility • Deploy at anytime • Release features independently • B ki Breaking change ‐> failed CD Objective h f il d CD Obj i • Invoke test suites of dependent components 34
  20. 20. Conclusion Conclusion • Continuous Delivery ensures that code is always ready  for release when the business wants to. y y • Quality is owned by the entire team. • Pipelines should be made as wide and as short as  possible • Quickly detect faults y p y • Teams confidently, rapidly roll 36