DevOps 101 - IBM Impact 2014


Published on

With Wai Lee, Vice President Prudential

Published in: Technology, Business

DevOps 101 - IBM Impact 2014

  1. 1. © 2014 IBM Corporation DevOps 101 Concepts and Overview Wai Lee Vice President Digital Platform Engineering Prudential Financial © 2013 IBM Corporation Sanjeev Sharma IBM WorldWide Lead – DevOps Technical Sales DevOps Blog: Twitter: @sd_architect Author: DevOps For Dummies –
  2. 2. Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
  3. 3. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  4. 4. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  5. 5. A lack of continuous delivery impacts the entire business enterprise in the new reality of “Systems Of Interaction” >45% of customers experience production delays >50% of outsourced projects fail to meet objectives >70% of budgets devoted to maintenance and operations 4-6 weeks to deliver even minor application changes to customers Systems of Interaction Continuous client experience Partner value chain Cloud-based Services Systems of Engagement Systems of Record Operations Rapid app releases impact system stability and compliance Suppliers Delivery in the context of agile Development/Test Speed mismatch between faster moving front office and slower moving back office systems, delaying time to obtain feedback Line-of-business Takes too long to introduce or make changes to mobile apps and services HR DB ERP MF iSeries CRM
  6. 6. William Deming – American statistician Major influencer of Japanese manufacturing and business Famous for Plan-Do-Check-Act cycle (Deming Cycle) PDCA cycles found in DevOps 5 William Edwards Deming Deming Cycle
  7. 7. DevOps approach: Apply Lean principles to software innovation and delivery to create a continuous feedback loop with customers Line-of- business Customer 1 3 2 1. Get ideas into production fast 2. Get people to use it 3. Get feedback Adopt DevOps approach to continuously manage changes, obtain feedback and , deliver changes to users Eliminate any activity that is not necessary for learning what customers want
  8. 8. 7 The Big Sources of Wasted Efforts: Find the Hidden Factory Overhead, rework, over-production Value-added production work 80% 20 % Lean Transformation Type of Waste Create Feature Deliver Feature Unnecessary Overhead Communicating ideas/knowledge Communicating between development and operations Unnecessary Re-work Tasks assigned back to developers from testing Tasks assigned back to developers from production rollbacks Over-production Unnecessary functionality produced Unnecessary hardware, data center, personnel 60% 40%
  9. 9. The Lean Adoption Framework Uncovers the Hidden Factory Opportunities Fat efforts to minimize Waiting Training Reporting Traceability Late rework Duplicate efforts Metrics collection Regression testing Change propagation Document generation Meetings/Checkpoints System administration Resource accounting Human inspections Streamline or automate More Valuable efforts to improve Scoping Learning Feedback Refactoring Designing Teaming Coding Testing Planning Engineering Empowering Prediction Deciding Steering Facilitate or smarten
  10. 10. Priorities of Indian Global System Integrators (200 responses) Fat efforts to minimize Late rework Waiting Regression testing Duplicate efforts Reporting Document generation Training Metrics collection Change propagation Traceability Human inspections Meetings/Checkpoints System administration Resource accounting Streamline or automate More Valuable efforts to improve Scoping Designing Planning Testing Reusing Deciding Steering Feedback Coding Prediction Engineering Learning Teaming Refactoring Facilitate or smarten
  11. 11. Enterprise capability for continuous software delivery that enables clients to seize market opportunities and reduce time to customer feedback Accelerate software delivery faster time to value Balance speed, cost, quality and risk increased capacity to innovate Reduce time to customer feedback improved customer experience IBM DevOps Continuous Release and Deployment Continuous Customer Feedback and Optimization Monitor & Optimize Develop & Test Release & Deploy Plan & Measure Continuous innovation, feedback and improvements Continuous Monitoring Collaborative Development Continuous Business Planning Continuous Testing
  12. 12. Systems of Engagement (SoE) Apps Rapid Releases AppStore Monitor and Optimize Release and Deploy Develop and Test IntegrationTest 11 Monitor and Optimize Develop and Test Web Apps Frequent Releases Production Environment Databases Systems of Record (SoR) Apps Fewer Releases Databases The need: Integrate systems of engagement with systems of record By bringing together the culture, processes, and tools across the entire software delivery lifecycle – spanning mobile to mainframe platforms
  13. 13. A Lean View of DevOps Idea/Feature/Bug Fix/ Enhancement Production Development Build QA SIT UAT Prod PMO Requirements/ Analyst Developer CustomersLine of Business Build Engineer QA Team Integration Tester User/Tester Operations Artifact Repository Deployment Engineer Release Management Code Repository Deploy Get Feedback Infrastructure as Code/ Cloud Patterns Feedback Customer or Customer Surrogate Metrics - Reporting/Dashboarding Tasks Artifacts
  14. 14. Demands on IT June 16, 2014 13 Limited amount of features and changes being deployed to keep pace with rapidly evolving business requirements NEW demand from business and technology BAU, Maintenance, Break Fix, Lights On Deliver systems of interaction for a consistent user experience across the enterprise Customer Experience Quality Agility Simplicity
  15. 15. DevOps June 16, 2014 14 Collaboration Communications Standards AGILE •Bridges not Silos •S.M.A.R.T. •Risk Management •Change and Release Management Policies and Procedures •Architecture •Continuous Integration •Continuous Delivery • Transform Legacy processes •Simplify workflows •Automate! •provide agility to keep pace •Audit and Compliance •Pre-approved Routine Changes •Service Catalog – Rapid Deployment and provisioning • Increase frequency of changes •Systems of Interaction/Innovation vs. Systems Record •smaller set of changes vs. large scale periodic releases •Do the things right vs. Do the right things •Tolerance of Risk for Development and Production •Engagement oriented teams •Specific Measurable Attainable Realistic Timely projects
  16. 16. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  17. 17. 16 • Common Business Objectives • Vision Statement • Common measures of Success • Building a DevOps Culture • There is no Silver Bullet • Right People are needed • Everyone is responsible for Delivery Product Owner Team Member Team Lead Team Member Team Member Senior Executives Users Domain Experts Auditors Gold Owner Support Staff External System Team Operations Staff Adopting DevOps in the Enterprise: People/Culture
  18. 18. • Organizational Change ‘Shift Left’ – Operational Concerns Adopting DevOps in the Enterprise: People/Culture •Do more upstream •App and infrastructure configuration management database •Dynamic Provisioning and Automated deployments •Automated testing – Test earlier and get feedback earlier •User testing early in QA and Staging. Not just Production. •Create test plans and test cases in Requirements Phase
  19. 19. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  20. 20. Orchestration End-to-End Software Delivery Process Provision Hardware •Virtual server configuration (S,M,L) •Operating System version Provision Server Instrumentat ion •Metering software •Monitoring software Provision Security •Groups •Service accounts •User accounts Provision Software •Provision Application Enabling •Software (AES) •Configure AES software •Archive and Catalog pattern image for reuse Produce App. Build •Build application version •Create install package •Create application configuration scripts Deploy App. Build •Install package •Run application configuration scripts Provision App. Data •Acquire application data and archive •Deploy seeded data •Run data configuration scripts (if any) I&O Processes I&O Processes w/ BU IT configuration/settings BU IT Processes
  21. 21. 20 • DevOps as a Business Process • A Process to get Capabilities from Ideation to Value • Apply Lean Thinking to Processes Adopting DevOps in the Enterprise: Process
  22. 22. DevOps Solution: Adoption paths and key capabilities Deployment Provisioning Release / Deploy Develop / Test Monitor / Optimize Monitoring Customer Feedback Code Test Portfolio Management Requirements Plan / Measure Change & Configuration Management Dashboards/ Analytics Business Owner Platform Operations Service Developer Collaboration Target Customer Jazz, OSLC and Open Standards Platform
  23. 23. Where do you start: DevOps Adoption Roadmap What am I trying to achieve? • Think through business-level drivers for improvement • Align vision and pains to common business drivers across silos • Look across silos, not just within the team, for improvements Where am I currently? • What do you measure and currently achieve • What don’t you measure, but should to improve • What practices are well scaled vs. incubating • Refine objectives to particular practice areas Where are my bottlenecks and priorities? • Business-level drivers expose practice gaps across silos • Focusing outside of the bottleneck limits overall improvement • It’s not just about tools, its about People, Practices, Technology, and information Step1Step2Step3 Current Practice Assessment Objective & Prioritized Capabilities Business Goal Determination Determine Activities Objective How should I plan my practice improvement? Step4 • Identify improvements to skills, processes, and tools to achieve desired outcome • Roadmap activity to define actionable plan • Target improvements which get the best bang for the buck Roadmap February 26, 2014 22
  24. 24. Understanding Maturity Levels Practiced Some teams exercise activities associated with the practice, inconsistently. No enterprise standards defined. Automation may be in place but without consistent usage models. Consistent Enterprise standards for practice are defined. Some teams exercise activities associated with the practice and follow the standards. No core team or COE to assist with practice adoption. Automation, if used, follows enterprise standards. Reliable Mechanisms exist to assist adoption and ensure that standards are being followed. Core team of mentors available to assist in adoption. Scaled Institutionalized adoption across the enterprise. COE is a matured and integral part of continuous improvement and enablement. Practices are mainstreamed across the enterprise. Feedback process in place to improve the standards. Specific maturity levels are defined by how well an organization can perform practices. The levels look at consistency, standardization, usage models, defined practices, mentor team or center of excellence, automation, continuous improvement and organizational or technical change management. February 26, 2014 23
  25. 25. Plan / Measure Development / Test Release / Deploy Monitor / Optimize ScaledReliableRepeatablePracticedPractice Based Maturity Model Define release with business objectives Measure to customer value Optimize applications Use enterprise issue resolution procedures Standardize and automate cross-enterprise Automate patterns-based provision and deploy Manage data and virtualize services for test Deliver and integrate continuously Link objectives to releases Centralize Requirements Management Measure to project metrics Link lifecycle information Deliver and build with test Centralize management and automated test Plan departmental releases and automate status Automated deployment with standard topologies Document objectives locally Manage department resources Manage Lifecycle artifacts Schedule SCM integrations and automated builds Test following construction Plan and manage releases Standardize deployments Monitor resources consistently Collaborate Dev/Ops informally Plan and source strategically Dashboard portfolio measures Monitor using business and end user context Centralize event notification and incident resolution Automate problem isolation and issue resolution Optimize to customer KPIs continuously Improve continuously with development intelligence Test Continuously Manage environments through automation Provide self-service build, provision and deploy Area of primary focus February 26, 2014 24
  26. 26. Key Capabilities 1. Collaborative Development & Continuous Integration 2. Continuous Business Planning 3. Continuous Release and Deploy 4. Continuous Testing 5. Continuous Feedback Adopting DevOps in the Enterprise: Process
  27. 27. Region Configuration – To Be Environment 26
  28. 28. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  29. 29. 1. Collaborative Development and Continuous Integration Mobile App Development Teams Enterprise Services Development Teams Adopting DevOps in the Enterprise: Process
  30. 30. Deployment Automation What one deploys may be anything from simple configuration changes; to incremental code changes towards a new feature; to Database schema changes; to changes to the environment; to the whole stack. Not just the application • Application Components • Middleware Configurations • Database Changes • Environment Configurations • Orchestrating the deployment process
  31. 31. Investing in Release Automation … Introducing UrbanCode Drive down cost Remove manual effort and wasted resource time with push button deployment processes Speed time to market Simple, graphical process designer, with built-in actions to quickly create deployment automation Reduce risk Robust configuration management, coordinated release processes, audits, and traceability Enabling clients to more rapidly deliver mobile, cloud, big data and traditional applications with high quality and low risk IBM UrbanCode Deploy automates the deployment of applications, databases and configurations into development, test and production environments, helping to drive down cost, speed time to market with reduced risk. IBM UrbanCode Release is an intelligent collaboration release management solution that replaces error-prone manual spreadsheets and streamlines release activities for application and infrastructure changes.
  32. 32. 31 Continuous Integration Architecture – Current State Jenkins 1.5 server Artifactory Server 3.0 Drop-Off PVCS Server Source Repository (SVN Server) 1 Devloper Checkin in SVN Server 2 Pull code From SVN and Display Result 3 Build source , create artifact(WAR , EAR , Jar , etc) 4 U pload Jarin Artifatory 5 . Check in file in PVCS 6.downloadfilefrom PVSforRelease 7. Copy file to Drop Off ACME Server Portal Server 8.CallAcme-Job forRelease 9.Pullfileform DROP- Off 10. Perform Release
  33. 33. 32 Provisioning Manager – Process Orchestration Network VIRTUALIZATION CPU MEMORY I/O & NETWORK AES OS Application VIRTUAL SERVER AES OS VIRTUAL SERVER 2b. Deploy Pattern 1b. Capture Pattern Provisioning Manager Configuration Database (Regions, Tiers, Applications) Pattern Repository Deploy Application Application Content (Data) Reporsitory Application Build packages Repository Build Master Process Process to clone DB Datafiles Application Build n,n+1,n-1 revisoins Seeded Application Data AD Security Provisioning AES OS AES OS 1a. Capture & Catalog Pattern Request 2a. Deploy Pattern Request 3a. Provision Security 4a. Deploy application build 5a. Deploy application data 4b.DeployBuild,RunAppconfiguration 5b.DeployData,RunDataConfiguration Source Code Library Management Application Databases 3b. Add service accounts, groups
  34. 34. Agenda • What is DevOps? • Adopting DevOps o People o Process o Technology • Implementing Delivery pipelines
  35. 35. Implementing a DevOps toolchain SCM Build / CI Server Unit testing Test Automation Test Stubbing Delivery Pipeline Environment Configuration Automated Monitoring Asset Repository
  36. 36. June 16, 2014 35 Continuous Delivery • Create a repeatable, reliable process for releasing software • Automate almost everything • Keep everything in version control • If it hurts, do it more frequently, and bring the pain forward • Instrument and Audit Everything • Test Everything • Everybody is responsible for the delivery process • Continuous improvement • Dynamic Infrastructure supports continuous delivery 35 Source: “Continuous Delivery: Reliable Software Releases Through Build, Test and Deployment Automation” Application Artifacts Application Code Instrumentation and Configuration Runtime Environment Definition Version Automate Test Track and Plan Instrument and Audit Dashboard Environment ApplicationConfiguration
  37. 37. Continuous Delivery Pipeline PVCS SVN Artifactory Splunk Jenkins Maven Ant Puppet chef HP ALM Policy Tester Selenium Smart Cloud Control Desk PMG ServiceNow Microsoft Team Foundation Server Serena Dimensions
  38. 38. Improve Change and Release Management 4-8-12-16-20 hours… Testing is 1/3 of our release time Lack of automation Too many hand offs 37 More environments Test earlier Automate Process Optimization Monitoring Continuous Service Improvement Testers Developers MiddleWare Admins DBA Ops Users Release Manager App Managers Are you done yet?! Why is it taking so long?! Where are we with the release?! When can I test?! Let’s fix it in production!
  39. 39. New “DevOps” team 38 Operations & Release Management Infrastructure Engineering Middleware Web, Security, Application Server Architecture Automation and Tooling
  40. 40. DevOps Tiger Teams 39 QA, TEST STRATEGY & PLANNING DEVELOPMENT BUILD PROCESS RELEASE & CHANGE GOVERNANCE & BUSINESS OUTCOMES CULTURE ELASTIC INFRASTRUCTURE • Automated monitoring & dash-boarding inclusive of business drivers & events • Implementation of application enabling software • Process for determining Capacity Planning and Capacity/Event Monitoring • Future scalability and how to measure • Orchestration process • Virtualization strategy across applications •Automation Strategy and Scope •Data Management •Baseline data for each application •After image baseline for each application •Test case transaction for each application. •Process controls & measures •Define Phase Gate/SDLC intersection •Changes/Implications to costing models based on dynamic virtualization •Knowledge Management •Define how to achieve the management and relationship between source, manifest, and executables
  41. 41. Questions?
  42. 42. We Value Your Feedback Don’t forget to submit your Impact session and speaker feedback! Your feedback is very important to us – we use it to continually improve the conference. Use the Conference Mobile App or the online Agenda Builder to quickly submit your survey • Navigate to “Surveys” to see a view of surveys for sessions you’ve attended 41
  43. 43. Thank You
  44. 44. Legal Disclaimer • © IBM Corporation and The Prudential Insurance Company of America 2014. All Rights Reserved • The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. • References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. • If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete: Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. • If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete: All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. • Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both. • If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete: Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. • If you reference Java™ in the text, please mark the first use and include the following; otherwise delete: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. • If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete: Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. • If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete: Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. • If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete: UNIX is a registered trademark of The Open Group in the United States and other countries. • If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete: Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. • If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.