Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated Deployment in Support of Continuous Integration to Transform SDLC


Published on

presentation in IBM Innovate 2013 in Orlando FL

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Automated Deployment in Support of Continuous Integration to Transform SDLC

  1. 1. WebMD Adopts Automated Deployment in Support of Continuous Integration to Transform their SDLC Teresa Dietrich, Vice President Technical Operations Derek Chang, Senior Director
  2. 2. We were here… 2
  3. 3. So we assembled a team… 3
  4. 4. To fix our SDLC… 4
  5. 5. And now everyone is happy. 5
  6. 6. Technology @ WebMD WebMD, Medscape, MedicineNet, eMedicine, UK cobrand Serving nearly 1 Billion Pageviews a month, 132 million unique Running over 200 separate applications, vast majority in-house developed Environments: Dev/Devint, QA01/02, QA00, Production/DR Two main data centers, geographically diverse OS: mix of Linux and Windows Datastores: Sql Server, Oracle, Mongo, Vertica and mysql Web: mix of Apache and IIS App: mix of Tomcat, ASP, .Net 3.x and 4.x Service: ActiveMQ, Memcache 6
  7. 7. Before - State of Deployments Build repository – a free for all NAS share; manual upload Package management – HP proprietary package library; manual upload/download Configuration management – HP proprietary CML Deployment module – HP software policy Deployment initiation – SBM ticketing system Pre/post deployment tasks – custom scripts and manual steps Dev Sends Email QA Opens Ticket Lead Assign Ticket Engineer Opens SubTickets DBA Runs Script Engineer deploys build QA Smoke Tests Ticket Closed 7
  8. 8. Before – Deployment Metrics year total # of work days average daily max date 2011 2650 251 10.5 34 7/1/2011 2012 3174 251 12.6 50 9/20/2012 Average time to deploy: 55 minutes 8% of tickets rejected thus required clarification Average build deployment per day: 11.6 Daily Max: 50 8
  9. 9. Before - Pain Points Require multiple systems Relies heavily on email for communication Inconsistent environment setup Only Ops did deployments in QA and Prod Error prone – Wrong information provided in ticketing system – Tons of customized scripts – Complexity of deployment procedures – Manual steps involved in pre and post deployment High maintenance cost – Configuration management – Host inventory – Deployment scripts 9
  10. 10. Becoming Agile Hired an Agile consulting group Signed up for Software to manage Agile Dev Trained Scrum Masters Learned to write user stories, plan sprints and used colored Post-Its Scheduled daily Stand-ups. Woohoo, now we are officially Agile! 10
  11. 11. Agile Implications for Deployments 11
  12. 12. Transform our SDLC 12
  13. 13. Waterfall vs Agile 13
  14. 14. Transform our SDLC 14
  15. 15. Promote Once vs Promote Continuously 15
  16. 16. Transform our SDLC 16
  17. 17. Static vs Elastic 17
  18. 18. Continuous Integration Team Formed a CI Team w/ representatives from: – – – – – Dev WebOps QA DBA Platform Ops Each team member had to have day to day hands-on responsibilities. Each team member had to have the authority to make decisions on behalf of the organization they represented. Each team member had to be able to commit to at least 4 hours a week . Team reported back to Sr Mgmt at specified milestones. 18
  19. 19. Measuring Success Automated Unit Testing Build Ready to Build Deployed (by env) Successful Build Deployment (by env) Successful Automated Smoke test (by env) Build Deployment Execution time (by env) Build from Dev to Production time 19
  20. 20. Automated Deployment Sr Management created a document with 1 page on assumptions and 1 page on milestones to kick off the team on their first assignment. Assumptions included: – Standards developed for: build, packaging, config, naming and monitoring – Roles and Responsibilities by Environment – Deployment success and Environment Exit criteria 20
  21. 21. Evaluation Process # Procedure Working group Management P1 principles and assumptions P2 define evaluation criteria X P3 define baseline functionalities and features collect list of products to be evaluated finalize product evaluation form and list to be evaluated X research products based on criteria review/rank products X X X P9 P10 review and finalize list of products for POC vendor demo In house POC X X X P11 review POC results X Description X P4 P5 P6 P7 P8 X X X referrals, forums X 5-7 products vendor documentation, forums and user community X POC should include must-have functionalities defined in F11 below 2-3 vendors 1 product at a time 21
  22. 22. Most Important Features Efficient configuration management Host inventory Capability of easy code promotion, backup and rollback Comparison – artifact/environment/configuration Orchestration engine – reusability and visibility Capability to integrate with other systems Support 22
  23. 23. Developing the Requirements Guidelines from Sr Management Review and understand the tools we were using. In-depth analysis of current deployment process Understand our current pain points. Learn from what products are available on the market. Learn from what other people are using and doing right. 23
  24. 24. Functional Requirements # Functionality Category importance F1 F2 start application stop application operation operation must-have must-have F3 application configuration management - administration operation must-have F4 F5 F6 F7 F8 F9 F10 application configuration management - versioning application configuration management - comparison application build deployment application build rollback fail-safe protection customizable post-deployment operations single interface for deployment-related operations administration administration operation operation usability operation usability must-have must-have must-have must-have must-have must-have must-have F11 log/report for troubleshooting and auditing administration must-have F12 F13 F14 F15 F16 integration administration administration Operation integration must-have must-have must-have must-have must-have integration with build server Visibility into deployment operation Event and hand-off Notification Scheduled Deployment Integration with configuration management solution 24
  25. 25. Functional Requirements # Functionality Category importance F17 integration with ticketing system F18 integration with revision control system integration integration nice-to-have nice-to-have F19 integration with software builder integration must-have F20 F21 F22 F23 F24 F25 F26 integration integration integration operation administration administration usability nice-to-have nice-to-have nice-to-have nice-to-have must-have nice-to-have must-have F27 deploy unbundled artifacts from multiple sources usability must-have F28 F29 F30 F31 usability integration administration usability nice-to-have must-have must-have must-have Integration with testing automation solution Integration with defect tracking solution Integration with availability monitoring database build deployment security and access control mechanism agentless architecture native support for scripting language mobile app deployment clustering database connectivity awareness resource inventory Support both windows/Linux platform 25
  26. 26. Evaluation Score Card Danny 3 Carrie 4 Matt 4 SRE 3 Ab Raj Yong total average Vendor support Vito 3 3 3 5 28 3.5 User community 5 4 4 5 4 4 3 5 34 4.25 Documentation 4 5 5 5 5 4 4 5 37 4.625 Cost for license* 1 2 2 1 1 1 2 2 12 1.5? Cost for infrastructure 2 2 3 3 2 2 1 2 17 2.125 Effort for maintenance 3 4 5 5 3 3 1 1 25 3.125 Cost for development and implementation 3 4 3 3 4 3 4 3 27 3.375 Interfaces such as API or web services 5 5 5 3 37 **4.625 Functionality Refer to table 1 5 4 5 4 5 5 5 5 38 4.75 Usability Ease of use 4 4 4 4 4 4 5 4 33 4.125 Learnability 4 4 3 3 4 3 5 4 30 3.75 Product support Cost (lower) Integrability 5 5 5 4 * including derived maintenance charge and yearly support ** scale will be adjusted prior to vendor evaluation phase 26
  27. 27. Necessary Standards and Adjustments Change of roles and responsibilities All configurations go into uDeploy and standards across apps. Standardize environment setup and rebuild if necessary. Standardize naming – Application/component/environment naming – Versioning and release number – Configuration key/value pairs – Agent/resource naming Automatic smoke testing 27
  28. 28. Identifying All Possibilities Source: referrals, communities, and conferences Up to 5 were proposed by each member 28 were identified as potential candidates 3 were given to each team member for initial screening 18 were eliminated for not meeting critical requirements 10 possible candidates moved to the second round 28
  29. 29. Choosing the Contenders Evaluation criteria: function checklist and vendor documents Priority ranked 10 remaining products Top 5 for on-site demo and discussion Each vendor was given criteria and use cases Use scorecard to rate and major pros/cons comparison Top 2 to participate on-site POC 29
  30. 30. Proof of Concept POC designed by WebMD Selected 2 most representative applications Success criteria – use cases and functional requirements Timeline – 5 days on-site and 2 weeks afterward Participated in installation, design and implementation Daily working session 30
  31. 31. Final Selection Use case verification and documentation In-depth review and discussion Pros and cons comparison The team voted unanimously and chose uDeploy! Evaluation result was communicated to Sr. Mgmt for final review On-site training for the entire Technology organization Demonstration along with highlights of the selection process 31
  32. 32. Demo 32
  33. 33. Key Benefits of uDeploy Usability, everything accessible in one tool Integration capabilities to existing SDLC tools Application Configuration Management Ease of promoting builds through environments Orchestration of 3rd party tools and reuse of templates Visibility – process, troubleshooting, reporting, approval and notification 33
  34. 34. After – Deployment Metrics Duration March/4/2013-June/3/2013 75 components developed – around 30% implemented 419 agents in use 1070 build deployments – 65 work days – 15.1 per day – Daily max: 43 on May/29/2013 Average - End to end time – 3 minutes 11 seconds 34
  35. 35. Lessons Learned Change is hard - redefining roles and responsibilities Bottleneck just shifted forward, on to the next challenge. DevOps culture - collaboration between Dev, Ops and QA. Ease of use, less technological expertise required When a piece of software is the centerpiece of your SDLC, it’s worth the work to find the right solution. Cross functional teams work well when they have clear direction and authority from management. Integration with other tools is key. Defining the process, policy and standards are as important as selecting the right tool. 35
  36. 36. And now how here we are…. 36
  37. 37. Appendix 1 – roles and responsibilities # SDLC environment Devint QA01/QA02 QA00 Production t1 t2 Environment Prep initiation (ticket or email to identify need) capacity evaluation/deployment target designation for new application Dev Dev/Ops QA Dev/Ops/QA-PERF QA Dev/Ops/QA-PERF Ops Dev/Ops/QA-PERF t3-1 t3-2 Dev Dev ProdOps ProdOps ProdOps ProdOps ProdOps ProdOps t3-3 t3-4 t3-5 t4-1 t4-2 t4-3 t4-4 t4-5 Server provisioning request database request mssql storage request DNS/VServer provisioning request middleware installation setup new uDeploy component template for applications update existing uDeploy component template for applications application related config variables/definitions app related config values***** middleware related config variables/values***** Dev Dev HPSA - ProdOps and dev self service Dev; signed off by Ops/QA Dev; signed off by Ops/QA Dev; signed off by Ops/QA Dev Dev ProdOps ProdOps ProdOps n/a n/a n/a QA ProdOps ProdOps ProdOps ProdOps n/a n/a n/a ProdOps ProdOps ProdOps ProdOps ProdOps n/a n/a n/a ProdOps ProdOps t4-6 t5 t6 t6-1 t6-2 t6-3 t7 snapshot approval for deployment people who push Deployment button Troubleshoot if deployment fails Automatic smoke testing Troubleshoot if smoke testing fails Testing and Sign-off snapshot implementation will be based on discussion with professional service Dev Dev Dev uDeploy/Selenium n/a Dev QA QA QA uDeploy/Selenium QA QA QA ProdOps ProdOps**** uDeploy/Selenium ProdOps**** QA Ops management ProdOps ProdOps**** uDeploy/Selenium ProdOps**** QA *template includes component and process templates, config key/value pairs **e.g. functional testing for QA01/02; UAT for QA00 ***based on promotion approach; will revisit during on-site training as well as professional service engagement ****party initiates communication and troubleshooting, which will involve Dev/Ops/QA ***** values will be contributed by all parties 37
  38. 38. Appendix 2 – provisioning process # 1 Setup uDeploy access** and perform uDeploy upgrade Step initiator CI 2 Setup devint based on Ops standard Dev 2.1 Provision new VMs - devint Deploy infrastructure software and servers 2.2 (IIS/Tomcat/Memcached/ActiveMQ) -devint 2.3 Provision DNS and Netscaler rules - devint owner PlatformOps ticket firstaid ticket#0 platformOps uDeploy ticket Dev SiteOps firstaid ticket#1 sysops host provisioning Dev ProdOps firstaid ticket#2 ProdOps general Dev ProdOps firstaid ticket#3 ProdOps general 2.4 Provision Storage - devint Dev StorageOps firstaid ticket#4 sysops storage provisioning 2.5 Provision DB - devint Dev DBOps firstaid ticket#5 DBOps general Dev Dev/ProdOps 3.1 naming standardization Dev/ProdOps Dev/ProdOps 3.2 Component mapping Dev/ProdOps Dev/ProdOps 3.3 Server inventory Dev ProdOps 3.4 Review/confirm server inventory Dev/ProdOps Dev/ProdOps 3.5 Install uDeploy agents Dev/ProdOps Sysadmin 3.6 Component/environment mapping Dev/ProdOps Dev/ProdOps 3 Application/component/environment population Dev/ProdOps Dev/ProdOps 4.1 Collect/Review/consolidate application config 4 Application config Dev/ProdOps Dev/ProdOps 4.2 Designate Config namespace Dev/ProdOps Dev/ProdOps 4.3 Import config Dev/ProdOps Dev/ProdOps 4.4 Build server configuration Dev Dev Dev Dev/ProdOps Dev Dev/ProdOps Dev Dev/ProdOps/DBOps Dev/QA/DBOps/ProdOps Dev/QA/DBOps/ProdOps QA firstaid ticket#6 sysops general Dev/ProdOps 5 Process Development 5.1 Develop Application Process if necessary Develop or Reuse component template (IIS/Tomcat/Windows 5.2 service/DB deployment,…,etc) Designate roles for each environment (configuration, deployment, 6 approval)*** 7 Set up additional notification if necessary; with product email alias ** provision user accounts and assign members to each privileged groups as shown in table 2 below *** admin for each environment will be responsible to designate roles 38
  39. 39. Appendix 3 - guidelines Creation of automation templates should be done at the rollout of the tool and not for each product or application. Types of templates may include Applications, Configurations and DB changes. Before a build is automated for the first time through the tool in DevInt, an existing template must be identified or a new template must be developed. If a new template is needed, there should an agreement that none of the existing templates are appropriate and involvement from a cross functional team to develop a new one. The Automated Deployment Process begins in the DevInt environment. The exit criteria from each environment should be at least one successful automated deployment using the prescribed tool. Success is defined as a successful unit test in DevInt or a successful smoke test in QA. Until the tool is identified, assumption is that the configuration lives with the build. When QA is referenced below, the assumption is that it is a member of the new Deployment Management team not a general QA analyst. We need to define, review and agree to a set of standards with regards to builds, build packaging, configurations, naming and monitoring to ensure the success of the automation deployment process. The groups listed below each environment have the responsibility of creating and executing the deployment and first level troubleshooting within that environment. 39
  40. 40. Appendix 4 – demo summary Application: – Quick summary of state of component/environment and configuration Component: – History view of who did what where. Environment: – Where you see version history and request details Host inventory Application property Component property Environment and environment-component property Global/system property: More to come – process, resource, version - properties/configuration management Application process – Manage component operations during deployment – sequence, lock, decision point and approval Component process – Where deployment actions take place – stop iis, check application pool, download artifact, search and replacement, start iis – Component process template Environment specific process Plugins Rest API Everything is versioned! Snapshot 40
  41. 41. Appendix 5-1: application view 41
  42. 42. Appendix 5-2: process template 42
  43. 43. Appendix 5-3: component view 43
  44. 44. Appendix 5-4: component process 44
  45. 45. Appendix 5-5: Component property 45
  46. 46. Appendix 5-6: component-environment view 46
  47. 47. Appendix 5-7: environment property 47
  48. 48. Appendix 5-8: environment inventory 48
  49. 49. Appendix 5-9: artifact view 49