An Agile DevOps Journey


Published on

Keynote presentation delivered at a March 13th event titled "Agility Across the Enterprise." The event was sponsored by BMC Software, Rally Software, and the Eliassen Group.

The presentation tells the story of a journey towards Agility from my own perspective working in BMC Software's IT Group. We were able to scale our productivity exponentially using the Agile methodology and DevOps practices & toolsets.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • About Me:Almost 4 years at BMCAlmost 10 years of IT experience – in small and large organizations – serving in a variety of rolesWorking primarily on implementing new products & platforms to help our Operational teams meet their goals – whether it’s our Back Office or Front Office groups.TODAY – I help lead the team responsible for serving the needs of our Marketing & Channel organizations
  • I’m here to tell you about a journey – that I was lucky to be a part of. A long and winding road that transformed the way we deliver value to our business. I’m going to talk about how we as an IT Department were just 3 years ago – what we implemented, the lessons we were able to take, and things we were able to accomplish. I’m NOT going to cite industry statistics, white papers, or Gartner stories. I’m going to talk from the trenches – what happened – how we were able to go from 4 releases per year – to 18 – with no increase in management overhead or no average increase in production defects.But First – I’m going to tell why you should care about all of this.
  • Let me explainwhy you care about the concepts of DevOps and AgilityWE DON’T MAKE WIDGETS HEREThe expression comes from the PERCEIVED inability to relate MANUFACTURING with ITIT is more complex –the work is mostly invisible – compared to manufacturing. And there are far more things that can go wrong.<NEXT 2 POINTS>HOWEVER----You are producing a SERVICE (sometimes a Product) – that is consumed by end usersTo deliver that SERVICE – you must perform a set of standard processes – you need:Parts / Inventory – aka INFRASTRUCTUREAssembly – aka DEVELOPMENTQuality AssuranceDistribution – aka DEPLOYMENTThey sound similar to me.To increase Customer Satisfaction, to decrease Cost, to reduce Errors / Outages, and to improve Time to Market – it becomes a question about Operational Efficiency and ThroughputSo that leads us to ….(next slide)
  • Increasing Resources isExpensiveOutsourcing means Productivity Delays – while the outsourcer learns your processes and business rulesCut back on Service or Product Catalogue – Question: How many people were thrilled to hear that the USPS isn’t going to deliver mail on Saturdays anymore?Rally – Manage DevBMC – Manage Ops
  • Critical mechanism of productivity is the job and materials release – our WIP was killing us. Disaster Recover Story:How does it make you feel to hear that 2 years ago – a company that makes and sells software for IT Departments – ran an entire weekend long Business Continuity (DR) Test using EMAIL and Microsoft Project?
  • Goal– push our teams to deliver more value – faster by combining the 2 product suites to overhaul our process
  • We implemented these 2 tool sets in step with the following principlesWIP – need transparency on work that is being doneLonger the time between Dev & End Users or Dev & Ops – means less feedbackMore conversations mean – getting it right the FIRST timeRepeating the same thing over and over is going to make you better at it Code in DEV is WORTHLESSRally – gives you the framework to increase visibility of WIP – what’s blocked, what’s Dev Complete, what’s Ready for Deployment.BMC – gives you the tools to increase repetition – Build & Deploy, Change Mgmt – which leads to mastery-All of this - drives down cost and reduces errors – leading to more frequent releases.If you want to learn more about these principles – Suggested Reading - The Phoenix Project
  • These buckets are in a logical orderRequirements Mgmt – allows us to attack the most important requests, get accurate estimates on how much it will take, and capture the reasons behind the requestSprint Planning – the estimates are used to derive the max. number of requests we can fit in within our cycle – and manage the tasks associated QA – all about transparency – standardizing what we test and ensuring that everyone knows what has been Accepted and ready to deployRelease Metrics – no more time spent having to dig up what we did each quarter – easy to export those details – and see who is accurately estimating work, etc.
  • Story about real-life February Release – Everything green – easy to see status (1st screenshot) – single pane of glass to manage roadblocks2nd Screenshot - Things get stalled (red) and may slip from the release3rd Screenshot But – we are able to have rich dialog on what is going on – questions or issues – from the same screen
  • 3 functional buckets of RLMRelease Planning – no more checklists – the plan can be constructed from existing templates (or create new ones) – and is one spot for both Dev & Ops to see. Graphically allowed to map out which steps have dependenciesRelease Execution – flexibility to have RLM execute scripts for deployment – and give you REAL TIME updates on what’s Not Started, Complete or BlockedRelease Tracking – accountability for tasks – requiring users to acknowledge steps to continue down the path – leaving room for Quality ImprovementRLM significant improves the Release by:Automating [How] the People Process [Who] Automating [How] the Packaging of the release components (application artifacts, environment changes, etc.) [What]Automating [How]the Deployment of the package to the appropriate environments [Where]
  • Key Points Current release processes cannot keep up with the increased complexity and deployment frequency of today’s applications Best practices combine multiple spreadsheets to create a “master checklist”The release process is chaotic compared to application development and Operational discipline – this is the DevOps Gap.Release Lifecycle Management is a Single Source of Truth for all application releases - CALENDAREmpowers teams with collaborative tools, Define the plan & map dependencies Create Templates for re-useRun the plan, Verify results, Improve for the next timeWatch/manage the progress in real time. ***No more solving complexity with complexityRLM is a critical link between Development, Operations, Change Management and release execution. It has enabled us to deploy our applications and releases faster, with fewer errors at the lowest cost.
  • Deployment – something every IT Department must deal with – they must do it – and they must do it wellContinuous UAT – dumping a whole set of changes at once on a customer means: 1) things will get overlooked 2) feedback is not going to be timely Story: A few years back – we were literally asking business people to commit to an entire month of testing!Retrospectives shouldn’t overlook what went well. Easy to focus just on improvement opportunities.
  • In order to deliver at this pace – we have to lay these products at where they fall in the lifecycle of software developmentBMC Help Desk is the front door for the End User to make their requestRally manages all the work that goes into the requestRLM organizes the steps to deploy the requestITSM / Remedy workflow manages the policy
  • 4 Releases  18 releases
  • Typically adding more resources to a team reduces their ability to deliver – with the tools and processes we had in place – we did not experience thisStory about Release PlanningEvery 6 months or so – stars & planets align to create the Perfect Storm – aka every team w/in IT has a deployment that weekendThis used to mean giving up at least 2 days for “Release Planning”
  • Transparency = more conversations – which leads to less rework – which means getting things right the FIRST timeThis makes IT + Business people HAPPYRepetitive Task Automation – Every deployment from an IT perspective has a “window” – or a time which the app is guaranteed to be UPThe last portion of any deployment is for any “smoke testing” or IT validation – when things go wrong – this is always the FIRST thing that is sacrificedAutomation means giving more time back to perform that quality checkSell Themselves – We are always acquiring new companies – most of the time there is work involved to identify the specific use case of how we can benefit from this toolThere is also usually a FORCING FUNCTION – to make sure that it is used NOT SO with RLM & Rally – the usage spread like wildfire - everyone just seemed to get it – ORGANIC Acceptance from the IT OrganizationSo much excitement that we have dedicated groups within IS&T just to gather feedback & best practices and relay to the product teams
  • An Agile DevOps Journey

    1. 1. Agility Across the EnterpriseBMC SoftwareChris Pearson, MBA, PMPIS&T – Front Office Automation
    2. 2. An Agile DevOps Journey We Don’t Make Widgets - or Why DevOps Matters “The Way We Were” What We Did What We Learned What We Accomplished© Copyright 3/13/2013 BMC Software, Inc 2
    3. 3. We Don’t Make Widgets? IT is a complex service Customers are being conditioned to expect demand more Departments are being asked to produce more with less resources© Copyright 3/13/2013 BMC Software, Inc 3
    4. 4. How Can You Keep Up? Increase Resources Explore Outsourcing Cut back on the Service or Product Catalogue Add Tools to Manage Your Process© Copyright 3/13/2013 BMC Software, Inc 4
    5. 5. “The Way We Were…” Unmanageable Backlog “Black Box” Release Scoping Spreadsheet-based Release Management No Accountability for Dev Estimates Back and Forth Email Discussion on Work Items “Start From Scratch” Deployment Plans Email Deployment Coordination Remember: We Are a Software Company (!)© Copyright 3/13/2013 BMC Software, Inc 5
    6. 6. Goal: Accelerate the change cycle to achieve more agility LOB / App Owner Plan Dev / App Ops / QA Support Build Run Release 6© Copyright 3/13/2013 BMC Software, Inc
    7. 7. DevOps Tenants WIP is the Silent Killer Long Release Cycles Decrease Feedback More Feedback Loops Yield Less Rework Repetition Leads to Habits Which Leads to Mastery Until Code is in Production, No Value is Generated© Copyright 3/13/2013 BMC Software, Inc 7
    8. 8. Rally – Application Lifecycle Management Applied Requirements Business Prioritization Estimation Management Value Sprint Task Timebox Planning Management User Quality Defect Test Cases Acceptance Assurance Tracking Testing Release Release Team Metrics Notes Productivity© Copyright 3/13/2013 BMC Software, Inc 8
    9. 9. Rally Screenshots© Copyright 3/13/2013 BMC Software, Inc 9
    10. 10. BMC – Release Lifecycle Management Applied Release Release DevOps Dependency Templates Collaboration Mapping Planning Release Build Task Automation Management Execution Release Audit / Quality Governance Improvement Tracking© Copyright 3/13/2013 BMC Software, Inc 10
    11. 11. BRLM Screenshots© Copyright 3/13/2013 BMC Software, Inc 11
    12. 12. What We Learned Deployment is a Fixed Cost - Automation is key to reducing delivery time and errors - All code must go through it Development and you cannot ship without • More Change! it Condition End Users for “Continuous UAT” - Deliver new functionality in a constant state, not “all at once” Operations Retrospectives are Critical • More Stability! - Guiding principle for continuous improvement© Copyright 3/13/2013 BMC Software, Inc 12
    13. 13. Software Development Lifecycle - SOLVED BMC - ITSM Rally - ALM BMC - RLM BMC - ITSM •Service Request • Requirements • Deployment Plan • Change Control Management • Development • Release Policy •Incident Estimates Automation • Impact Analysis •Problem • Quality • Release Assurance Coordination •Work Order • Sprint Planning & Execution Operations Development DevOps Operations© Copyright 3/13/2013 BMC Software, Inc 13
    14. 14. What We Accomplished BMC RLM Implemented 18 16 14 Releases – Per Year 12 Rally Implemented 10 8 6 4 2 0 2010 2011 2012© Copyright 3/13/2013 BMC Software, Inc 14
    15. 15. Behind the Numbers Average Number of User Stories – Per Release - 2010: 18 - 2012: 20 Frequency of Release - 2010: 3 months - 2012: 2-3 weeks Total Number of User Stories Released - 2010: ~65 - 2012: ~320© Copyright 3/13/2013 BMC Software, Inc 15
    16. 16. Behind the Numbers (continued) Headcount Efficiencies - NO  Decrease in team productivity  Increase in management overhead  Increase in post-production defects Release Planning Effort - 2010: 5+ 1 hr Meetings for 6 people (average) - 2012: 1 1hr Meeting for 6 people (average) Release Window (Outage) - 2010: 18 hours - 2012: 12 hours© Copyright 3/13/2013 BMC Software, Inc 16
    17. 17. Key Takeaways Transparency is KEY to increasing Throughput - Developers - Dev and Ops - Dev and End Users Automate Repetitive Tasks - Deployment Automation allows more time for Quality Deployments The right mix of tools will sell themselves within your organization© Copyright 3/13/2013 BMC Software, Inc 17
    18. 18. Learn more at© Copyright 3/13/2013 BMC Software, Inc 18