Massive Continuous Integration and Light-speed Iterations

415 views

Published on

Continuous integration (CI) has become a buzzword, with most engineering organizations claiming they've adopted the practice. However, the sad truth is that unreliable tests, long feedback loops, and poor configuration management block their efforts and minimize CI's potential benefits. Jesse Dowdle shares how AtTask radically redesigned its engineering pipeline and, through massive CI scaling, drove three days of testing to just minutes. Learn the pros and cons of different CI systems and how to integrate them with the cloud. Watch a live demo of AtTask's internal test and CI systems, which they’ve designed to make "Every commit a potential release candidate"-meaning that every commit is an iteration. Arm yourself with the talking points to sell massive CI to executives. Learn how, by leveraging publicly available tools, you can deliver hyper-scalability to your continuous integration framework and drive cycle time, defect counts, and technical debt to new lows.

Published in: Technology
  • Be the first to comment

Massive Continuous Integration and Light-speed Iterations

  1. 1.           BT3 Concurrent Session  11/8/2012 10:15 AM                "Massive Continuous Integration and Light-speed Iterations"       Presented by: Jesse Dowdle AtTask, Inc.                 Brought to you by:        340 Corporate Way, Suite 300, Orange Park, FL 32073  888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Jesse Dowdle AtTask, Inc. Senior manager of development for AtTask Inc. Jesse Dowdle directs the engineering efforts of nearly a dozen agile teams in the U.S. and overseas. Successful engineering in a fast-paced industry requires constant research in emerging technologies. Jesse drives thought leadership on continuous delivery, development best practices, and process evolution to help AtTask ship software at lightning speed. Happiest when he’s building great software, Jesse sees technology, process, and people as equally important prerequisites for success. A native of Utah, where he enjoys fast cars in the summer and snowboarding in winter, Jesse tweets at @dowdlemj and blogs.
  3. 3. dowdlemj@attask.com Massive Continuous Integration and Light-speed Iterations •Jesse Dowdle, Director of Development •@dowdlemj •linkedin.com/in/jessedowdle •www.continuousdeliverist.com
  4. 4. •Salt Lake City, Utah •Software •Project as a Service Management •Collaborative •Salt Work Management Lake City, Utah •Software •Project as a Service Management •Collaborative Work Management
  5. 5. •Salt Lake City, Utah •Software •Project as a Service Management •Collaborative Work Management Agile at AtTask 5 years Wall to wall 11 Dev Teams
  6. 6. How we did Scrum Sprint Sprint Sprint How we did Scrum 4 WEEKS Sprint Sprint Sprint
  7. 7. How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Sprint Sprint How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Sprint Public Release Sprint Public Release Public Release
  8. 8. How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Internal Internal Sprint Internal Public Release Internal Internal Sprint Internal Public Release Internal Internal Internal Public Release Internal Public Release How we did Scrum 2 days Demo Retrospective Plan 4 WEEKS Sprint Internal Internal Sprint Internal Public Release Internal Internal Sprint Internal Public Release Internal Internal
  9. 9. Then Scripted scenarios Slow, brittle Monthly Releases 3-5 days acceptance Manual administration Now Tight unit-like UI tests Fast, Robust Daily releases 30-45 minutes Sign-off and go
  10. 10. What is Continuous Integration? “Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.” -- Martin Fowler Components •Version •Build •Test Control System Server Runner •Test Suite •Notification Mechanism
  11. 11. Basic Integration •Version •Build •Test Control triggers new action Server retrieves latest version, compiles, starts Application Stack Runner executes tests against Application Stack •Results •Build are collected and sent by notification Server is cleaned and restored. Basic Integration
  12. 12. BASIC INTEGRATION BUILD SERVER VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
  13. 13. BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
  14. 14. BUILD SERVER APPLICATION STACK VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION BUILD SERVER VERSION CONTROL TEST RUNNER NOTIFICATION SYSTEM BASIC INTEGRATION
  15. 15. Massive Test Automation •800 Unit tests •3300 API tests •2000 UI tests •55,000 hours per month •Release Acceptance + CI “True” Continuous Integration •How often do I integrate? •What tests tell me I’m integrated? •What must I know to release?
  16. 16. Pipeline Priorities •Performance (Cycle Time) •Scalability •Visibility •Accountability ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
  17. 17. Commit ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Commit Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
  18. 18. Selenium Grid Grid + AtTask Commit Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Selenium Grid Grid + AtTask Commit Install Build Installer Build Tests ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY
  19. 19. Selenium Grid Grid + AtTask Commit Install Build Installer Build Tests Unit Integration 1/4 3/4 2/4 4/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 2/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY 3/4 4/4
  20. 20. Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 ATTASK’S PIPE BUILD -- DEPLOY -- TEST -- DESTROY HOW TO MAKE A PAPER AIRPLANE OR, QUEUEING THEORY AS EXPRESSED BY A FOLDED PROJECTILE Tear Down
  21. 21. Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Tear Down Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 SERIAL VS PARALLEL Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 2/4 SERIAL VS PARALLEL 3/4 4/4 Tear Down
  22. 22. CYCLE TIME CYCLE TIME
  23. 23. CYCLE TIME CI Vendors CruiseControl Jenkins (Hudson) TeamCity Go Team Foundation Server
  24. 24. Cloud Vendors Amazon Web Services OpenStack SauceLabs JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN
  25. 25. JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN JENKINS DYNAMIC SLAVE ALLOCATION
  26. 26. JENKINS DYNAMIC SLAVE ALLOCATION SLAVE JENKINS DYNAMIC SLAVE ALLOCATION
  27. 27. SLAVE SLAVE SLAVE SLAVE JENKINS DYNAMIC SLAVE ALLOCATION JENKINS DYNAMIC SLAVE ALLOCATION
  28. 28. Scalability •Taking •To the Test Suite to massive scale the cloud with Amazon Cloud Formations •Dynamic Selenium Grids •Dynamic AtTask Environments Divide and Conquer •Module separation will allow for scale •Test Suites should support sharding •Run tests with different dependencies in parallel
  29. 29. Divide and Conquer Selenium Grid Selenium IE Grid + AtTask Commit Selenium FF Install Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 Divide and Conquer Unit Integration 1/4 3/4 2/4 4/4 Tear Down
  30. 30. PARALLEL BY DEPENDENCIES In Memory Database Email PARALLEL BY DEPENDENCIES UI Only Customer Data
  31. 31. In Memory Database Email UI Only Customer Data PARALLEL BY DEPENDENCIES From 3 days to 30 minutes with Selenium 2
  32. 32. SELENIUM 1 (1800 TESTS) SOMETIMES THE TESTS NEED TO CHANGE SELENIUM 1 (1800 TESTS) 4 HOURS SLOW SOMETIMES THE TESTS NEED TO CHANGE
  33. 33. TRANSITIONING TO SELENIUM 2 TRANSITIONING TO SELENIUM 2
  34. 34. TRANSITIONING TO SELENIUM 2 SELENIUM 1 (1800 TESTS) 4 HOURS TRANSITIONING TO SELENIUM 2
  35. 35. SELENIUM 1 (1800 TESTS) 4 HOURS SELENIUM 2 (750 TESTS) 30 MINUTES TRANSITIONING TO SELENIUM 2 TRANSITIONING TO SELENIUM 2 WHERE DID THE TESTS GO?
  36. 36. 2.1 MINUTES TRANSITIONING TO SELENIUM 2 2.1 MINUTES TRANSITIONING TO SELENIUM 2 34 SECONDS
  37. 37. 2.1 MINUTES 34 SECONDS WITH GRID... TRANSITIONING TO SELENIUM 2 2.1 MINUTES 34 SECONDS WITH GRID... 8 SECONDS TRANSITIONING TO SELENIUM 2
  38. 38. 2.1 MINUTES 34 SECONDS WITH GRID... 8 SECONDS 2.4 SECONDS TRANSITIONING TO SELENIUM 2 JSON PARTLY CLOUDY WITH A CHANCE OF RAIN
  39. 39. JSON PARTLY CLOUDY WITH A CHANCE OF RAIN Scalability Selenium Grid Selenium IE Grid + AtTask Commit Install Selenium FF Build Installer Build Tests Other tests Unit Integration 1/4 3/4 2/4 4/4 Tear Down
  40. 40. Scalability Selenium Grid Grid + AtTask Commit Build Installer Build Tests SLAVE IT’S RAINING GRIDS
  41. 41. SLAVE IT’S RAINING GRIDS SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS
  42. 42. SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS SLAVE SELENIUM GRID SELENIUM GRID IT’S RAINING GRIDS
  43. 43. SLAVE SELENIUM GRID SLAVE SLAVE SELENIUM GRID SLAVE SLAVE IT’S RAINING GRIDS Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
  44. 44. Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
  45. 45. Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health Visibility •Test results in Jenkins •Description •Theres •View •All plugin my commit! plugin the jobs on one line, oh the humanity! •Screen-shotting •Test UI Tests Age, Project Health
  46. 46. Accountability •Claim / Blame test results using AtTask Accountability •Claim / Blame test results using AtTask
  47. 47. Accountability •Claim / Blame test results using AtTask CI as a Platform •On-Demand •Easy demo environments button for version control •Visibility •Other into production systems quality measurements •Static Analysis •Code Coverage
  48. 48. Staffing for Massive CI •Get the right skill set •Watch your integration points •Technological •Organizational •DevOps Adoption •Process / Tool alignment •Information •Integration •Scrum Radiators with existing tools board •Backlog •Integration with Release Management
  49. 49. CI EVOLUTION ITS A CONTINUOUS INVESTMENT What does it cost? •3 FTE •Two human, one machine(s) •Keeping •Efficient costs down machine sizes •Rock-solid •Elasticity tear down
  50. 50. Return on investment •5 days (40 hours) to certify one release... •10 engineers •$40 per hour •$20,000 •30 minutes to certify one release... •10 engineers •$40 per hour •$250 To release 12 times in a year... $240,000 TO CERTIFY $3000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI To release 24 times in a year... $480,000 TO CERTIFY $6000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI To release 100 times in a year... $2,000,000 TO CERTIFY $25000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI
  51. 51. 100 deployments annually $2,000,000 100 deployments annually $265,000 $2,000,000
  52. 52. 100 deployments annually $265,000 Where to go from here •Continuous Delivery •Application Lifecycle Management •Continuous Integration of Infrastructure •Open Source
  53. 53. Tools Manifest •AtTask •EC2 - contributions... wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin •CloudFormation •Pipeline View - •Description •GIT - - wiki.jenkins-ci.org/display/JENKINS/AWS+Cloudformation+Plugin github.com/JoelJ/PipelineDashboard Setter - github.com/JoelJ/DescriptionSetter /wiki.jenkins-ci.org/display/JENKINS/Git+Plugin •AtTask for Jenkins - Coming soon... Further Reading •“Continuous Integration” -- Martin Fowler - http://www.martinfowler.com/ articles/continuousIntegration.html •Extreme Programming Explained -- Kent Beck - http://amzn.to/c0VvTB •“Continuous Deployment in 5 Easy Steps” -- Eric Ries - http:// radar.oreilly.com/2009/03/continuous-deployment-5-eas.html •“Four Principles of Low Risk Software Releases” -- Jez Humble - http:// www.informit.com/articles/article.aspx?p=1833567 •“Continuous Delivery” -- Jez Humble & David Farley - http:// continuousdelivery.com/ •The Lean Startup -- Eric Ries - http://theleanstartup.com/ •Continuous Integration -- Paul Duvall - http://amzn.to/9Zx1l1
  54. 54. Thank you! •Jesse Dowdle, Sr Manager of Development •@dowdlemj •linkedin.com/in/jessedowdle •www.continuousdeliverist.com •David Tolley, Sr QA Automation Engineer •@davidtolley •linkedin.com/profile/view?id=81007074

×