Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM InterConnect 2016 Greg Hodgkinson 2238 Thriving DevOps at BMI (Prolifics)


Published on

Hear how a thriving DevOps team has helped boost delivery at BMI.

Published in: Software
  • Be the first to comment

  • Be the first to like this

IBM InterConnect 2016 Greg Hodgkinson 2238 Thriving DevOps at BMI (Prolifics)

  1. 1. CONNECT WITH US: IT: Customized to Your Advantage Thriving DevOps at Broadcast Music Inc. DOD-2238 Growing from nothing to world class. Public | Copyright © 2014 Prolifics Greg Hodgkinson – Prolifics, Global Director of Lifecycle Tools and Methodology Jim Harvey - Broadcast Music Inc., Sr. Dir. of Quality Assurance and Release Management
  2. 2. CONNECT WITH US: What attendees will learn  What has made DevOps thrive at Broadcast Music Inc.?  What are the conditions and driving forces behind a thriving DevOps practice?  What does this thriving practice look like?  Who is involved, what tools are they using, how have they benefitting from these tools?  What are they achieving?  How is DevOps making a difference to the business? 2
  3. 3. CONNECT WITH US: IT: Customized to Your Advantage Introductions Public | Copyright © 2014 Prolifics
  4. 4. CONNECT WITH US: Who Are We Making Happy? 4 Everyone! (that listens to music)
  5. 5. CONNECT WITH US: Speaker Introduction 5 JIM HARVEY Senior Director of Quality Assurance and Release Management Broadcast Music Inc. Jim Harvey is the Senior Director of Quality Assurance and Release Management for Broadcast Music Inc. (BMI). He has over 16 years’ experience in Software Development. Jim is currently evolving the Software Development Life Cycle at BMI as it relates to Requirements Management, Software Testing and Change Control management. GREG HODGKINSON Director of Lifecycle Tools and Methodology Prolifics Greg has 20 years of experience in assisting customers achieve success by optimizing their teams. Optimizations include adopting agile practices, DevOps tools and practices, and SOA techniques. He is very much a practitioner with responsibility for successful delivery at a number of leading companies. He presents on DevOps and agile SOA approaches at major IBM events, has co-authored a Redbook on SOA solutions, and contributes to DeveloperWorks.
  6. 6. CONNECT WITH US: Broadcast Music Inc. at a Glance  Broadcast Music, Inc. - 1939  Performing Rights Organization (PRO) - Pay public performance royalties  Operate on a non-profit-making basis  7 locations: Nashville, New York, Los Angeles, Atlanta, Miami, Puerto Rico, London  600 employees, 7.5 million works, over 500,000 songwriters and composers 6
  7. 7. CONNECT WITH US: The Performers 7 Israel Kamakawiwoʻole Louis Armstrong Judy Garland
  8. 8. CONNECT WITH US: The Writers 8 “What a wonderful world” George David Weiss Bob Thiele “Somewhere over the rainbow” 1939 1967
  9. 9. CONNECT WITH US: 2016: The Digital Music Ecosystem  Over 500 digital music services worldwide  Offer consumers the opportunity to legally access up to 26 million songs  Challenge: Stay relevant and competitive? 9
  10. 10. CONNECT WITH US: Prolifics at a Glance 10 5-Years Compound Annual Growth Rate 19% Employees Worldwide 1,500 Global Presence United States, United Kingdom, Germany, India 20+Technology Accelerators 550+ Technical Certifications Over 10 Technology and Solutions Awards since 2009 including Business Agility, Customer Integration and Digital Experience, the first ever Beacon Laureate for Business Agility Over 160 global customers are currently Fortune 1000 companies Best-in-class architects and specialty experts: BPM, Integration, Digital Experience, Security, Testing, Business Analytics and Enterprise Content Management End-to-End Project Expertise Rate of Repeat Engagements* 91% Years in Business 35+ Offices 14 Awards Technology Expertise Fortune 1000 *based on % revenue Source – December 2013 internal revenue metrics
  11. 11. CONNECT WITH US: IT: Customized to Your Advantage Challenge and Success 11/14/2016 11
  12. 12. CONNECT WITH US: Why the need for DevOps at BMI?  Rapidly innovating music industry  Means BMI needs rapid innovation themselves  New music services => new integrations  New offerings = > new systems  Explosion of artists => increased need for scale ========================== = Net: Need to deliver a lot of software innovation, quickly. 12
  13. 13. CONNECT WITH US: The BMI Transformation  Over a period of 4-5 years BMI have fundamentally changed their ability to deliver software innovation. 13 New Approaches New Runtime Platforms New Development Tools Agile BPM SOA DevOps IBM BPM IBM WebSphere Portal IBM ODM WebSphere DataPower IBM Integration Bus WebSphere ESB IBM CastIron IBM DataStage Oracle IBM WSRR Focal Point DOORS NG Rational Software Arch. Rational Team Concert Rational Quality Mgr. IBM UrbanCode Deploy
  14. 14. CONNECT WITH US: How Did They Change (Successfully) So Fast?  They key to this rapid transformation: Good processes and tools! 14 To be successful with these… …required that they be successful with these. In a nutshell: all the things that helped them be efficient! DevOps especially relevant.
  15. 15. CONNECT WITH US: Adding Agile and RTC  First challenge: Herding the cats!  Scrum  Structure for teams  Focus of sprints  Course correction  Simple – get going quickly!  RTC  Progress visible  Keeps everyone on same page  Easy configuration management  Opportunity for integrated builds 15
  16. 16. CONNECT WITH US: Toe in the Water: Continuous Integration  Opportunity: Integrated builds for quality feedback loop  Continuous integration of WESB apps  Java apps easy, start tackling newly adopted WESB  Developed build machine plugin to automate builds of WESB ears  Used in CI cycles: immediate feedback for devs (learning new tech)  Little effort -> lots of value 16
  17. 17. CONNECT WITH US: Obvious Next Step: Automated Deploys  Challenge: Delays to get changes into test  Continuous deployment of WESB apps  Added automated deployment of WESB ears to plugin  Either on demand, or as soon as ready  No more waiting; testers testing earlier  Reduced delays mean more testing done; quality increases 17
  18. 18. CONNECT WITH US: Expand Across Enterprise Architecture  Opportunity: Success breeds success  Full-stack build and deploy  Made use of Prolifics investment  Common framework  Leveraged common steps  Already had plugins for IIB and Portal  Expanded to all app types  Integrated stack no longer held back by slowest individual release pipeline 18
  19. 19. CONNECT WITH US: Added Database Deployments  Challenge: But what about the data?  New plugin for automating deploy of schema updates  Implemented using same framework and standards as apps for simplicity  DBAs place scripts into source control  Automation picks up and upgrades schema  Works across all environments 19
  20. 20. CONNECT WITH US: Added Publishing of Services  Challenge: And what about the service registry?  New plugin to publish to WSRR  Again, same framework and standards leveraged for simplicity  Uploads WSDLs, XSDs, and other file changes  Updates version and other metadata  Consistent across all environments  One less thing to remember to do 20 *Bad luck Brian
  21. 21. CONNECT WITH US: Ramping up on Automated Deployment  Challenge: Push button deployments – too many buttons!  Purpose-built deployment engine: UrbanCode Deploy  Wrap existing deployment automation as UCD plugins - quickly and painless move  Orchestration  Cross-tier deployments simple  Easier to elaborate deploys of individual components too  Snapshots of complex apps simple  Conditional gates on environments 21
  22. 22. CONNECT WITH US: Integrated Automated Testing  Opportunity: Include existing automated tests  Plugin to wrap testing tool(s)  Tests moved into source control  Builds pull out tests and execute them automatically  Can then  Trigger when you need,  Schedule,  Or include as part of build/deploy/test process  Lots more value from test scripts  Regression testing  Smoke testing 22
  23. 23. CONNECT WITH US: Dockerising Build Agents  Challenge: Build infrastructure inflexible time sink  Created Docker images of all build agent configurations  Install build agent software  Install automation  Install supporting software – in some cases this is considerable  Easier to isolate build agents  Easier to spin up new agents  Easier to refactor infrastructure 23
  24. 24. CONNECT WITH US: Next: Automated Provisioning of Environments  Challenge: Full set of test environments takes up a lot of tin  Create Docker images for all test environment configurations  Saves time – also allows new behaviors  Spin up new environments on demand  Stand-up/tear-down as part of test automation  Much better usage of tin – easier to share across projects and teams 24
  25. 25. CONNECT WITH US: Beyond That: Into the Future  Opportunity: What else can we do with this automation?  Think big for the future!  Increasingly hands-free automation  Self scaling environments  Detect shortage  Provision new environment  Lay on top app and config  Self healing environments  Detect issue  Provision different version  Apply known fix 25
  26. 26. CONNECT WITH US: IT: Customized to Your Advantage Our Approach 11/14/2016 26
  27. 27. CONNECT WITH US: Practice Agility!  The key to getting started and keeping going. 27 Some might… We did… Start by envisioning a big program of work. Started simple – Then build on initial success, now into 4th year of continued improvement – best multi-year projects are based on a small success. Static delivery plan Good principles of agility – Do something, course correct, adapt to changing needs, focus on strategic plan but still keep agility to respond to immediate need Fix-it-all in first 6 months Pick an area to improve - Don’t “boil the ocean”. We started by just fixing Continuous Integration.
  28. 28. CONNECT WITH US: Focus on Practices and Benefits!  Focus on what is important. 28 Some might… We did… Just focus on putting in place new tools and technologies. Consider the practices – We thought through what it is we’re trying to do and chose the right tools and technologies to support. Get caught up in “what is hot in DevOps”. Focus on benefit – As we intended on keeping resource/budget small, we had to make sure that everything we did gave “best bang for buck”. Practices & Benefits
  29. 29. CONNECT WITH US: Cross Boundaries!  Important to keep eyes open to wider opportunities. 29 Some might… We did… Stick with the areas they are comfortable with e.g. Java. Cover it all! - Thrived on figuring out how to automate assembly and deploy of the trickier areas e.g. database, repository- based dev, service registries. Ensure that all parts of the app are included. Just focus on developers and operations teams. Consider all involved teams – Any role that is involved in releasing apps, including DBAs, testers.
  30. 30. CONNECT WITH US: Keep Efficient!  DevOps sounds like it needs a big team. 30 Some might… We did… Go it alone, or outsource it all. Best of both worlds – Prolifics involvement allows BMI to focus on writing and testing apps, while BMI release engineering team has still had direct control of app releases. Create a big team to solve a big problem. Small as possible - Achieved almost unbelievable levels of efficiency with team of just 2.5 FTEs (and one of those is a manager). Low cost, disproportionate benefits. Either do no training, or overcommit to training. Pragmatic value-driven training - Train by example, support with JIT workshops and Wiki- based documentation.
  31. 31. CONNECT WITH US: Leverage Expertise!  Working with a partner has allowed BMI to make much more progress. 31 Some might… We did… “Not invented here!” – lets build our own automation. Reuse! - Leveraged automation from the Prolifics DevOps Labs. Huge difference in how much has been achieved as well as level of quality. Give us 6 months and we’ll have our own automation development capability. Engage specialists - Prolifics DevOps labs efficiencies have allowed us to be responsive to BMIs needs – we’ve created the engine.
  32. 32. CONNECT WITH US: IT: Customized to Your Advantage What Does Success Look Like? Public | Copyright © 2014 Prolifics
  33. 33. CONNECT WITH US: Planning and Prioritizing Work  Apply agility to planning  Start from backlog  Release plans track what we’re currently working on  Thrice weekly touchpoints  Focus on prioritizing:  Hot current issues  Hot opportunities for improvement  Strategic improvements  Constantly maximizing value! 33
  34. 34. CONNECT WITH US: Prototyping New Automation  Uncover risk, uncover opportunity  Seek to understand before we automate  What are artifacts? Where do artifacts live? What are APIs/tools?  Approach  Do it manually. Then automate manually – command-line. Then script automation  Examples!  Involve experts and teams - Demo and get feedback 34 DeveloperWorks Knowledge Center Samples Try it all out!
  35. 35. CONNECT WITH US: Implement Automation in RTC  Great automation development support  Configuration management of automation and example apps  Inject automation (with changes) into example build workspace  Build engine to do test runs – attach logs, perform activity logging  Link up development work to plan items in release plan  Record of progress, issues, solutions  JBE simple automation engine – use Ant as standard 35
  36. 36. CONNECT WITH US: Architect for Ease of Maintenance and Extensibility  Well factored plugin model  Ant-based core  Standard assembly and deploy steps atomic steps  Additional atomic steps e.g. stop app, undeploy app, set default app, etc.  Composite steps are mini-processes  Reusable utilities in common  Package up libs where appropriate  Helper scripts where appropriate  Interface with APIs and tools appropriate to target platform  Version to handle upgrades of target platforms 36 APIs and Tools Target Platform Core Automation Plugin build.xml libs scripts Atomic •assembleApp •deployApp •etc. Composite •assembleAndDeployApp common.xml
  37. 37. CONNECT WITH US: Wrap Plugins for UrbanCode Deploy  Port deploy automations to enterprise deployment platform  Simple set of wrapper files around proven automation  Benefits  Keep orchestration (deploy processes) flexible  One button deploys for multi- component, multi-tier apps  Snapshot whole app and move through environments  Gates, approvals, etc. 37 UCD Plugin Core Automation Plugin build.xml libs scripts Atomic •assembleApp •deployApp •etc. Composite •assembleAndDeployApp common.xml plugin.xml wrapper scripts.groovy Atomic • deployApp • etc. component template.json • component processes
  38. 38. CONNECT WITH US: Continuous Integration and Provisioning of Plugins  Practice what we preach – optimized delivery of plugins  Automate plugin assemble and deploy  Core plugins and UCD plugins  Reduce time-to-test  Run in a continuous integration cycle to detect issues early  Combine with running of automation steps to test working  Feedback loop  Snapshot plugins – promote to PROD 38 DEV/TEST PROD DEV/TEST Deploy PROD Deploy DEV/TEST Deploy PROD Deploy JBE UCD Server J B E J B E J B E UCD Server
  39. 39. CONNECT WITH US: Containerize build agents  Creating a flexible DevOps infrastructure with Docker  Build agent host installs take time  Docker gives simple way of  Codifying install process  Shipping around “golden” images  Starting/stopping as many instances (containers) as you wish  Keeping build machine stacks isolated  Provisioning new machines “instantly”  Prevent build machines from clogging up with installs and temporary files 39 Before: After:
  40. 40. CONNECT WITH US: IT: Customized to Your Advantage How Has This Helped? Public | Copyright © 2014 Prolifics
  41. 41. CONNECT WITH US: What has been achieved?  Automated assemble and deploy across 10 technology platforms.  Accelerated delivery pipelines for 11+ application development streams.  Oversight of deployments for all involved projects.  Core Build/Release team of 2.5 FTEs 41
  42. 42. CONNECT WITH US: What Benefits are Being Seen?  Able to reinvest upwards of 10 resources into more valuable activities.  Delivery is sooner.  Testers are able to test more.  Quality of code delivered is higher.  Visibility of delivery has increased.  Outages due to deployment error close to eliminated. 42
  43. 43. CONNECT WITH US: Impact on the Business, Customers, and Us?  Able to keep up with exploding number of music delivery channels.  Able to deliver more features that make it easier to do business.  Giving a better and more reliable quality of service to customers.  Collected royalties result in a healthier music industry, meaning better music for all of us! 43
  44. 44. CONNECT WITH US: Thank You Your Feedback is Important! Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.
  45. 45. CONNECT WITH US: Contact Us Public | Copyright © 2016 Prolifics 45 +1 646 435 2470