Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data-Driven DevOps: Mining Machine Data for 'Metrics that Matter' in a DevOps Workflow

1,514 views

Published on

IT organizations are increasingly using machine data - including in DevOps practices - to get away from 'vanity metrics' and instead to generate 'metrics that matter'. These metrics provide visibility into the delivery of new application code and the business value of DevOps, to both IT and business stakeholders.

Machine data provides DevOps teams and others - including QA, secops, CxOs and LOB leaders - with meaningful and actionable metrics. This allows stakeholders to monitor, measure, and continuously improve the velocity and quality of code throughout the software lifecycle, from dev/test to customer-facing outcomes and business impact.

In this session Andi Mann, chief technology advocate at Splunk, will share core methodologies, interesting case studies, key success factors and 'gotcha' moments from real-world experience with mining machine data to produce 'metrics that matter' in a DevOps context.

Published in: Technology
  • Be the first to comment

Data-Driven DevOps: Mining Machine Data for 'Metrics that Matter' in a DevOps Workflow

  1. 1. Copyright © 2016 Splunk Inc. Mining Machine Data for ‘Metrics that Matter’ in a DevOps Workflow
  2. 2. Abstract (Hidden) IT organizations are increasingly using machine data – including in DevOps practices – to get away from ‘vanity metrics’ and instead to generate ‘metrics that matter’. These metrics provide visibility into the delivery of new application code and the business value of DevOps, to both IT and business stakeholders. Machine data provides DevOps teams and others – including QA, secops, CxOs and LOB leaders – with meaningful and actionable metrics. This allows stakeholders to monitor, measure, manage, and continuously improve the velocity and quality of code throughout the software lifecycle, from dev/test to customer-facing outcomes and business impact. In this session Andi Mann, chief technology advocate at Splunk, will share core methodologies, interesting case studies, key success factors and ‘gotcha’ moments from real-world experiences with mining machine data to produce ‘metrics that matter’ in a DevOps context.
  3. 3. DevOps is a Culture of Empathy & Sharing INTEGRATION COLLABORATION COMMUNICATION BETWEEN DEV AND OPS TO DELIVER BETTER SOFTWARE, FASTER METHODS FOR IMPROVING
  4. 4. Shared Feedback Enables ‘The Three Ways’ Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”
  5. 5. Empowered DevOps Teams Empathy - more than understanding • Feel your teammates’ pain • Understand their work and your impact Empowerment - more than making decisions • Be responsible in decisions, activities • Be accountable to your team of teams
  6. 6. DevOps Workflow is Becoming Complex and Opaque 6 Build (Jenkins, Bamboo) Code (Git, MS-TFS) Plan (Jira, Rally) Test/QA (Cucumber, SonarQube) Stage (Pivotal, AWS) Release (Jenkins, Octopus) Data Center Device Data Engagement Data Config (Puppet, Ansible) Monitor (NewRelic, Dynatrace) Cloud Services Network Services www/HTTP Data Social Sentiment Wire Data Application Data Continuous Integration (CI) / Continuous Delivery (CD) Site Reliability Engineering Business Impact Monitoring API ServicesSecurity/Compliance
  7. 7. DevOps complexity raises risk of failure ● Slower Speed ● Longer MTTR ● Lower Quality ● Reduced Agility ● Poor Visibility ● Hard to Scale ● Increased Waste ● Impaired Collaboration 7 DevOps From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega, http://www.gartner.com/document/3096018
  8. 8. From every tool, every process, every component, on-prem or off The One Constant: Machine Data
  9. 9. Common Data Fabric 9 API SDKs UI Other Tools Escalation/ Collaboration Visibility Across the Whole Dev Lifecycle Plan Code Build Test/QA Stage Release Config Monitor
  10. 10. Common Data Fabric 10 API SDKs UI Server, Storage. N/W Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Other Tools Ticketing/Help Desk Custom Applications Visibility Across the Whole Ops Environment API Services
  11. 11. Machine Data From DevOps Tools 11
  12. 12. Provisioning and Config Metrics 12
  13. 13. Machine Data from QA/Pre-Prod/Staging 13
  14. 14. Machine Data from Release Servers 14
  15. 15. Machine Data from Infrastructure Systems 15
  16. 16. Machine Data from Database Servers 16
  17. 17. Machine Data from Customer Systems
  18. 18. What do you measure and why?
  19. 19. I’m working super hard!! That’s my stapler.
  20. 20. 20 Yeah, but … … what are you achieving? I’m gonna need you to come in Sunday.
  21. 21. 21 Daily Active Users? Installs? Downloads? Sales?
  22. 22. DevOps Metrics that Matter Culture e.g. • Retention • Satisfaction • Callouts Process e.g. • Idea-to-cash • MTTR • Deliver time Quality e.g. • Tests passed • Tests failed • Best/worst Systems e.g. • Throughput • Uptime • Build times Activity e.g. • Commits • Tests run • Releases Impact e.g. • Signups • Checkouts • Revenue
  23. 23. Gartner’s DevOps ‘Metrics that Matter’ Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay
  24. 24. IDC’s DevOps ‘Metrics that Matter’
  25. 25. What Are Your ‘Metrics That Matter’?
  26. 26. Finding Your Metrics That Matter Work from business backwards Mine realtime machine data Close the feedback loops 26
  27. 27. Outcomes
  28. 28. Measurement drives Feedback loops Velocity Deliver on time & on budget IT is delivering on time, on budget IT and Business Leaders Impact Deliver code for business needs IT is achieving business goals IT and Business Leaders, Customers, Staff Show you when you deliver. And when you don’t. Quality Deliver the quality you promised We deliver a quality experience for users Dev and Ops Organizations
  29. 29. Measurement identifies ‘Waste’ Plan Develop (UI) Develop (Db) Develop (M’ware) Develop (Backend) Security Test Monitor Build (Prod) Architect Secure/ Comply DeployAccept Unit Test Document Cap Plan Train Feedback Integration Test Configure System Test Launch CAB Develop (APIs) Budget Build (Dev) Mgmt/ Tooling W W W W W W W W W 16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12
  30. 30. Measurement Ensures Transparency • Release when ready, not a date! • Best / worst developers • Best / worst providers • Impact of new code on ops • Impact of new code on biz
  31. 31. Measurement Enables Continuous Improvement Defect Information Capacity Planning Quality Standards Enhancement Requests Integration Requirements Acceptance Metrics Service Levels and KPIs Application Development Test and Acceptance Production BuildCodePlan Test/QA Stage Release Config Monitor Infrastructure Dependencies
  32. 32. Measurement Improves Quality Code quality scans Static security scans White BoxDeveloper checks in code Automated Acceptance Tests Dynamic Security Scans Black Box “Chaos Monkey” tests Test Fail: Return Test Fail: Return X X Production QA Prod Pattern QA Pattern Library Test Pass: Promote Test Pass: Promote to Production Pattern library used for test and QA
  33. 33. Measurement Accelerates Velocity Pivot & improve with Continuous Insights Product Managers identify new opportunities Continuously delivered to market … and Auditors are “happy”
  34. 34. Measurement Aligns Business Impact
  35. 35. Fast-feedback loop for actionable commercial insights So You Can Innovate at Market Speed BUSINESS DEV/OPS CUSTOMERS HOW IS OUR: • Security? • Quality? • Stability? • Performance? • Compliance? HOW IS OUR: • Market Launch? • Feature Usage? • Marketing Changes? • Prioritization? • Customer Sat?
  36. 36. Summary
  37. 37. Metrics that Matter Drive Better Feedback Loops Improve Application Velocity Visibility across silos, tools, and processes exposes bugs and bottlenecks so you can remediate, iterate, and innovate faster. Improve Application Quality Track quality across multiple teams, tools, systems, and service providers, so you can find and fix more issues before production Improve Application Impact Real-time analytics correlates application delivery with business goals, so you can drive better experience and iterate faster
  38. 38. Sources/Additional Reading ● splunk.com/DevOps - Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc. ● dev.splunk.com – Resources for developing with or on ther Splunk platform, incl. SDKs, API Docs, guides, etc. ● blogs.splunk.com – Check the ‘DevOps’ and ‘Ansible’ tags for specifics, including how to deploy Spunk w/ Ansible ● splunkbase.splunk.com – Splunk add-ons and applications incl. Ansible Tower App for Splunk and 1000+ more ● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016 ● 2016 State of DevOps Report, DevOps Research and Assessment ● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis ● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford ● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay ● Metrics that Matter, Mark Michaelis, IntelliTect ● DevOps and the Cost of Downtime: Fortune 1000, IDC ● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014 38
  39. 39. Copyright © 2016 Splunk Inc. Thank You! Andi Mann Chief Technology Advocate, Splunk @andimann

×